Nov 01 22:22:54.078938 ip-10-0-138-155 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:22:54.542294 ip-10-0-138-155 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:54.542294 ip-10-0-138-155 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:22:54.542294 ip-10-0-138-155 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:54.542294 ip-10-0-138-155 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:54.547569 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547458    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.547569 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547570    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.547569 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547575    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547578    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547581    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547584    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547587    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547589    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547593    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547596    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547598    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547601    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547604    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547606    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547609    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547611    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547614    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547617    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547619    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547623    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547625    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.547671 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547628    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547631    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547634    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547636    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547639    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547641    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547644    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547659    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547661    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547664    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547669    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547672    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547675    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547678    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547681    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547684    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547686    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547689    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.548121 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547691    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547694    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547697    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547699    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547702    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547704    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547707    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547709    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547712    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547714    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547717    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547719    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547722    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547724    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547727    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547729    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547732    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547738    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547742    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.548608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547745    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547748    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547751    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547754    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547756    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547759    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547762    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547765    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547767    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547770    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547772    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547775    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547778    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547781    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547783    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547786    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547788    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547791    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547793    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547796    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.549048 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547799    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547801    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547804    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547806    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547808    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547811    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547814    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547816    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547818    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547821    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547823    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547826    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547828    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547831    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547834    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547836    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547839    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547843    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547848    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.549522 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547852    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547855    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547857    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547860    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547863    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547865    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.547868    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549795    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549807    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549811    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549817    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549820    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549823    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549825    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549828    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549830    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549833    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549836    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.549984 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549839    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549841    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549844    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549846    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549849    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549851    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549854    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549856    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549858    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549861    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549864    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549867    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549869    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549872    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549874    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549878    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549881    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549883    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549886    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549888    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.550454 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549891    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549894    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549896    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549898    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549901    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549903    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549906    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549909    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549911    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549914    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549916    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549919    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549921    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549924    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549926    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549928    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549931    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549934    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549937    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549939    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.551003 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549941    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549944    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549947    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549950    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549952    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549955    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549957    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549960    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549963    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549966    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549968    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549971    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549974    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549976    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549979    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549982    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549985    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549989    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549993    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549996    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.551501 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.549999    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550001    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550004    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550007    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550009    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550012    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550014    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550017    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550019    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550024    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550028    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550031    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550033    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550036    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550039    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550042    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550044    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550047    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550049    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.552030 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550052    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550055    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550058    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550060    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550063    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550065    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550068    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550070    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550073    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550075    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550078    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550080    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550083    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550085    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550088    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550627    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550633    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550639    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.552695 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550642    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550645    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550649    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550652    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550656    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550659    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550662    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550665    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550667    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550670    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550673    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550676    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550679    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550683    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550685    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550688    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550691    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550695    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550699    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.553329 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550702    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550705    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550708    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550710    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550713    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550715    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550718    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550720    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550723    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550725    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550728    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550730    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550733    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550735    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550738    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550740    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550742    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550745    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550747    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550750    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.553808 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550753    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550755    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550758    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550761    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550763    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550766    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550769    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550772    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550775    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550777    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550780    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550782    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550785    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550787    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550790    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550792    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550795    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550797    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550800    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550802    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.554304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550805    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550807    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550809    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550812    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550815    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550818    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550820    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550823    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550825    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550830    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550832    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550835    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550837    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550840    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550842    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550845    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550848    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550851    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.554798 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550853    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550856    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550859    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550861    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550864    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550866    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550869    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550871    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550874    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550876    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550879    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550881    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550884    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550886    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550888    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550891    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550893    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550896    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550899    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550901    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.555253 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550903    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550906    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550908    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.550911    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.551984    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552000    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552007    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552012    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552017    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552021    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552027    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552031    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552035    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552038    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552042    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552045    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552048    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552051    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552054    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552057    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552060    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552063    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552068    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552071    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:22:54.555772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552074    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552077    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552081    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552085    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552088    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552091    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552094    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552097    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552100    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552103    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552106    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552109    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552113    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552116    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552119    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552122    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552125    2577 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552129    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552134    2577 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552190    2577 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552378    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552386    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552392    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552400    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:22:54.556327 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552406    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552412    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552418    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552423    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552434    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552439    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552444    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552450    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552454    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552459    2577 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552466    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552471    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552477    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552486    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552491    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552497    2577 flags.go:64] FLAG: --help="false"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552502    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552507    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552512    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552517    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552523    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552550    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552556    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:22:54.556918 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552562    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552566    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552572    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552577    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552582    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552588    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552593    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552603    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552607    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552612    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552617    2577 flags.go:64] FLAG: --lock-file=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552623    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552627    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552632    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552641    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552650    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552655    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552660    2577 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552665    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552670    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552675    2577 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552680    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552687    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552692    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552704    2577 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:22:54.557460 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552711    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552716    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552720    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552725    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552730    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552735    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552740    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552758    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552764    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552769    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552773    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552778    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552782    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552788    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552798    2577 flags.go:64] FLAG: --port="10250"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552803    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552807    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08cba2557fe8b4eb1"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552813    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552818    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552823    2577 flags.go:64] FLAG: --register-node="true"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552828    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552835    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552840    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552849    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552854    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552860    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:22:54.558053 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552865    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552870    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552875    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552880    2577 flags.go:64] FLAG: --runonce="false"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552885    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552895    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552900    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552904    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552909    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552914    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552919    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552931    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552936    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552941    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552952    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552957    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552963    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552968    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552973    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552983    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552988    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.552998    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553007    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553013    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553017    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:22:54.558683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553022    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553027    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553033    2577 flags.go:64] FLAG: --v="2"
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553039    2577 flags.go:64] FLAG: --version="false"
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553051    2577 flags.go:64] FLAG: --vmodule=""
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553057    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553063    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553485    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553495    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553501    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553504    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553507    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553509    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553512    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553516    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553519    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553522    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553524    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553530    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553533    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553549    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.559304 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553552    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553555    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553558    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553561    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553563    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553566    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553568    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553571    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553574    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553576    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553579    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553581    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553584    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553587    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553590    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553592    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553595    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553597    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553600    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.559871 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553610    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553613    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553615    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553618    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553621    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553623    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553626    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553630    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553633    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553637    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553641    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553646    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553649    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553651    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553654    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553657    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553660    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.560341 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553662    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553665    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553667    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553670    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553672    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553675    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553677    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553680    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553682    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553684    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553687    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553689    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553692    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553694    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553696    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553699    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553701    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553704    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553707    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553710    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.560806 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553713    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553715    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553719    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553722    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553724    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553726    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553733    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553735    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553738    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553740    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553742    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553745    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553748    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553750    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553753    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553755    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553758    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553761    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553763    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.561287 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553766    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553768    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553771    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553773    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553776    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553779    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553781    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553784    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553786    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553790    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553794    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553798    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553804    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.553836    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553883    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.561765 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553889    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553911    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553915    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553921    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553925    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553930    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553933    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553936    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553938    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553941    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553943    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553946    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553949    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553952    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553957    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553959    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553962    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.562144 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553964    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553967    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553969    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553972    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553975    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553977    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553979    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553982    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553984    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553987    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553990    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553993    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553995    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.553999    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554002    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554004    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554008    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554011    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554013    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554016    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.562608 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554018    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554021    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554024    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554026    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554029    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554032    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554035    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554037    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554040    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554043    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554045    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554048    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554050    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554054    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554058    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554061    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554064    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554066    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554069    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.563080 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554071    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554074    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554076    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554079    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554082    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554084    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554089    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554092    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554094    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554099    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554101    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554104    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554106    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554109    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554111    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554114    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554116    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554119    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554121    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554124    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.563531 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554126    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554129    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554131    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554134    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554136    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554139    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554141    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554144    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554146    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554149    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554151    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554153    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554156    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554158    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554161    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554163    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554166    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554169    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554173    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.564145 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554176    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554178    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554182    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554185    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554188    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554190    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554192    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.554196    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.561843    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.561863    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.563480    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.563487    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563565    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563572    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563575    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563578    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563582    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563585    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563587    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563591    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.564641 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563594    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563597    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563599    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563602    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563604    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563607    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563610    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563613    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563615    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563618    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563620    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563623    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563626    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563628    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563631    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563633    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563636    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563638    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563641    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563645    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.565115 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563648    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563653    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563656    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563659    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563661    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563664    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563666    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563669    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563672    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563675    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563677    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563680    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563682    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563685    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563688    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563690    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563693    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563695    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563698    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563700    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.565590 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563703    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563706    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563708    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563711    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563713    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563716    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563718    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563721    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563723    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563726    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563728    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563731    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563734    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563737    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563740    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563743    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563746    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563750    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563754    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563758    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.566060 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563761    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563763    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563766    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563769    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563771    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563774    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563776    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563778    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563781    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563784    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563786    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563789    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563791    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563794    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563796    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563799    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563803    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563807    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563810    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.566532 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563813    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563815    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563818    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563820    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563823    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563826    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563829    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563832    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563835    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563837    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563840    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563843    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563845    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563848    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563850    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563853    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563855    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.566991 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.563861    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.563896    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564045    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564049    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564052    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564055    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564058    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564060    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564063    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564066    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564068    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564071    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564073    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564076    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564079    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564081    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.567409 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564084    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564086    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564089    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564092    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564094    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564097    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564099    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564102    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564105    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564107    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564110    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564113    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564116    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564118    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564120    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564123    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564125    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564128    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564130    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.567878 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564133    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564135    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564137    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564140    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564142    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564145    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564147    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564150    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564153    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564156    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564158    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564163    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564166    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564169    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564172    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564175    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564178    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564180    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564184    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.568382 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564186    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564189    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564192    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564195    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564197    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564200    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564202    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564205    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564207    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564210    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564212    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564215    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564217    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564220    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564222    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564225    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564227    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564230    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564232    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564235    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.568843 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564237    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564240    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564243    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564245    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564248    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564251    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564253    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564256    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564258    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564261    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564263    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564267    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564270    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564273    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564275    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564278    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564281    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564283    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564286    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564288    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.569433 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564291    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564293    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564295    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564298    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564300    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564303    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564305    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564308    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564310    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564313    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564315    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564317    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564320    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564322    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.564346    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564380    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:54.569928 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564388    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564391    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564394    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564397    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564400    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564403    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564405    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564408    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564411    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564413    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564416    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564418    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564421    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564424    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564427    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564429    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564431    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564434    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564436    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564439    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:54.570497 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564444    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564446    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564449    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564451    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564454    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564456    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564459    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564461    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564464    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564466    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564469    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564473    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564476    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564478    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564481    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564483    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564485    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564488    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:54.571288 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564491    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564493    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564496    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564498    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564501    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564504    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564507    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564509    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564512    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564514    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564517    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564519    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564523    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564527    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564530    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564532    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564551    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564554    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564557    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:54.571793 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564559    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564562    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564565    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564567    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564570    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564572    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564577    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564579    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564582    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564584    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564587    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564590    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564593    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564595    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564598    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564600    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564602    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564605    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564610    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:54.572474 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564613    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564616    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564618    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564621    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564623    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564626    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564628    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564631    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564633    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564635    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564638    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564640    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564644    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564648    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564653    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564656    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564658    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564661    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564668    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:54.573344 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564670    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564672    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564675    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564678    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564680    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564683    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:54.564685    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.564757    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.567907    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.568946    2577 server.go:1014] "Starting client certificate rotation"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.569057    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.569118    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.571139    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:54.573870 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.572296    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:54.586573 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.586555    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:22:54.589454 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.589431    2577 log.go:25] "Validated CRI v1 image API"
Nov 01 22:22:54.590267 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.590246    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:22:54.597720 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.597691    2577 fs.go:135] Filesystem UUIDs: map[6cf40e16-d200-447e-80c5-16ce70ac3c77:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89b5243b-12d2-4764-bc56-c896d0f01c4f:/dev/nvme0n1p3]
Nov 01 22:22:54.597720 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.597716    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 22:22:54.603334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.603202    2577 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:54.601779235 +0000 UTC m=+0.391335997 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101008 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c638dbd86fb64791b01c4aeb93c1c SystemUUID:ec2c638d-bd86-fb64-791b-01c4aeb93c1c BootID:548210c3-255a-4abe-a307-6a236f6516c6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fd:92:8f:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fd:92:8f:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:34:88:29:d8:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:22:54.603334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.603324    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:22:54.603469 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.603421    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:22:54.604736 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.604692    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:22:54.604884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.604739    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-155.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:22:54.606260 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.606246    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:22:54.606302 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.606263    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:22:54.606302 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.606277    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:54.606302 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.606287    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:54.607435 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.607420    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:22:54.611479 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.611463    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:22:54.611625 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.611616    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:22:54.612947 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.612935    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:22:54.612994 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.612952    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:22:54.612994 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.612969    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:22:54.612994 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.612978    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:22:54.612994 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.612986    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:22:54.613350 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.613338    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:54.616252 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.616234    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:22:54.622173 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.622153    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:54.623032 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623017    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:22:54.623115 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623057    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:22:54.623263 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623250    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:22:54.623324 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623270    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:22:54.623324 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623280    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:22:54.623324 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623289    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:22:54.623324 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623297    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:22:54.623324 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623306    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:22:54.623941 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623931    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:22:54.623991 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623947    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:22:54.623991 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623960    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:22:54.623991 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623977    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:22:54.624101 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.623997    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:22:54.624101 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.624012    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:22:54.624957 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.624946    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:22:54.625017 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.624962    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:22:54.629076 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.629058    2577 server.go:1263] "Started kubelet"
Nov 01 22:22:54.629187 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.629145    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:22:54.629254 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.629202    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:22:54.629295 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.629280    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:22:54.629878 ip-10-0-138-155 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:22:54.630571 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.630436    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:22:54.632262 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.632245    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:22:54.636448 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.636417    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 22:22:54.637577 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.637555    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:22:54.637681 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.637562    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:54.637736 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.637699    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:22:54.638274 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.638253    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:22:54.638274 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.638255    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:22:54.638412 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.638285    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:22:54.638412 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.638410    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:22:54.638499 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.638418    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:22:54.640986 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.640966    2577 factory.go:55] Registering systemd factory
Nov 01 22:22:54.640986 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.640988    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:22:54.641306 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.641290    2577 factory.go:153] Registering CRI-O factory
Nov 01 22:22:54.641306 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.641308    2577 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:22:54.641428 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.641380    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:22:54.641510 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.641492    2577 factory.go:103] Registering Raw factory
Nov 01 22:22:54.641581 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.641522    2577 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:22:54.644699 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.644677    2577 manager.go:319] Starting recovery of all containers
Nov 01 22:22:54.654875 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.654858    2577 manager.go:324] Recovery completed
Nov 01 22:22:54.658805 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.658792    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.660927 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.660904    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.660975 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.660946    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.660975 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.660959    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.661505 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.661487    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:22:54.661505 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.661502    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:22:54.661639 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.661516    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.661639 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.661522    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:22:54.662335 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.662321    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.662420 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.662351    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.662420 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.662361    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.663818 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.663804    2577 policy_none.go:50] "Start"
Nov 01 22:22:54.663818 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.663819    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:22:54.663896 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.663832    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:22:54.665948 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.665936    2577 policy_none.go:44] "Start"
Nov 01 22:22:54.705319 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.705274    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:22:54.707786 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.707770    2577 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:22:54.708380 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.708359    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:22:54.708454 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708383    2577 server.go:87] "Starting device plugin registration server"
Nov 01 22:22:54.708735 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708723    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:22:54.708786 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708740    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:22:54.708858 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708837    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:22:54.708958 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708944    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:22:54.709005 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.708958    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:22:54.709510 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.709491    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:22:54.709626 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.709532    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:22:54.764479 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.764426    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-155.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 22:22:54.764479 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.764425    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-155.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 22:22:54.766492 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.764453    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-155.us-west-2.compute.internal.18e40c06b9eb8437  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-155.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-155.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:54.628856887 +0000 UTC m=+0.418413643,LastTimestamp:2026-11-01 22:22:54.628856887 +0000 UTC m=+0.418413643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-155.us-west-2.compute.internal,}"
Nov 01 22:22:54.784495 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.784451    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:22:54.785859 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.785840    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:22:54.785914 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.785868    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:22:54.785914 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.785890    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:22:54.785997 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.785924    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 22:22:54.809529 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.809449    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.810379 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.810356    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.810501 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.810406    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.810501 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.810420    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.810501 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.810446    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.854073 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.854029    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:54.854225 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.854110    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tlsfr"
Nov 01 22:22:54.886475 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.886438    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"]
Nov 01 22:22:54.886642 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.886510    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.887470 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.887453    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.887470 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.887482    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.887627 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.887492    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.890370 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.890340    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.890567 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.890530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.890646 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.890591    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.891194 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891175    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.891283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891211    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.891283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891220    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.891283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891243    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.891283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891252    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.891283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.891225    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.893391 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.893375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.893479 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.893402    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:54.894134 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.894117    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:54.894238 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.894144    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:54.894238 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.894158    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:54.913074 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.913043    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.915095 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:54.915078    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.926357 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.926334    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:54.928386 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.928366    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:54.939661 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.939632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.939749 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.939666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:54.939749 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:54.939684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.029446 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.029416    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:55.030229 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.030210    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.030311 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:55.030245    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-155.us-west-2.compute.internal\": node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:22:55.030676 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.030662    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:55.038228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.038204    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.039892 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.039953 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.039953 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.040017 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039979    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.040017 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039993    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.040017 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.039981    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.149308 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.149241    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:55.150030 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.150016    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.217056 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.217014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.219153 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.219136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:22:55.249884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.249721    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:55.568791 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.568768    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 22:22:55.569334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.568894    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 22:22:55.569334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.568910    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 22:22:55.569334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.568948    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 22:22:55.569334 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:55.568931    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a40c4aa6d7453489f8ffaf27c6ab6881-5876fe58ba2f5954.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-155.us-west-2.compute.internal.18e40c06bbd56bbd\": read tcp 10.0.138.155:53012->77.113.24.206:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-155.us-west-2.compute.internal.18e40c06bbd56bbd  default   4826 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-155.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-155.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-155.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:54 +0000 UTC,LastTimestamp:2026-11-01 22:22:54.810424147 +0000 UTC m=+0.599980902,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-155.us-west-2.compute.internal,}"
Nov 01 22:22:55.613160 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.613112    2577 apiserver.go:52] "Watching apiserver"
Nov 01 22:22:55.638691 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.638667    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:55.859436 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.859331    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:55 +0000 UTC" deadline="2028-07-05 08:33:45.672877993 +0000 UTC"
Nov 01 22:22:55.859436 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.859371    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14674h10m49.813511175s"
Nov 01 22:22:55.863513 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.863490    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:22:55.863829 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.863812    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-sdh5p","openshift-network-diagnostics/network-check-target-7flsn","openshift-network-operator/iptables-alerter-w67hg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj","openshift-image-registry/node-ca-46rjc","openshift-multus/multus-additional-cni-plugins-2prlz","openshift-multus/multus-nsn92","kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-v5fjs","kube-system/global-pull-secret-syncer-n7thn","kube-system/konnectivity-agent-hxbq6","openshift-cluster-node-tuning-operator/tuned-q7rnx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal","openshift-dns/node-resolver-x57zp"]
Nov 01 22:22:55.866227 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.866212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:55.866318 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:55.866281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:22:55.868258 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.868240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:55.868343 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:55.868292    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:22:55.870347 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.870332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:55.872505 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.872490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.874605 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.874589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:55.876693 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.876678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.878837 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.878821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.881939 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.881917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.884121 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.884102    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:55.884217 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:55.884167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:22:55.886211 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.886194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:55.888404 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.888374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.890599 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.890576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:55.922547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.922523    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:22:55.922713 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.922527    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:55.922713 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.922511    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:55.922713 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.922609    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:22:55.924280 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.924259    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:22:55.924421 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.924353    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:22:55.924476 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.924422    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:22:55.924518 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.924492    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:22:55.926508 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.926491    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:22:55.926508 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.926505    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:22:55.926624 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.926550    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:22:55.926624 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.926582    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:22:55.928941 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.928921    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:22:55.929049 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.928945    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:22:55.929111 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.929082    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:22:55.929158 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.929110    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:22:55.929158 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.929118    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:22:55.939034 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.939015    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:22:55.945765 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.945898 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.945898 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.945898 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.945898 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946031 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946031 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.946031 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.945992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946031 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946031 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946228 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxbq\" (UniqueName: \"kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgwh\" (UniqueName: \"kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjcd\" (UniqueName: \"kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.946899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.946991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mzv\" (UniqueName: \"kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh77z\" (UniqueName: \"kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947277 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2qn\" (UniqueName: \"kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccjg\" (UniqueName: \"kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.947745 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjsw\" (UniqueName: \"kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:55.948244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:55.948244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:55.948244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:55.948244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:55.948244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.947766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxt2k\" (UniqueName: \"kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:55.974913 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.974888    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:55.975059 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:55.974940    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-chrc5"
Nov 01 22:22:56.032334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032300    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:22:56.032334 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032322    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:22:56.032514 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032373    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:22:56.032671 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032655    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:22:56.032747 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032731    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:22:56.032802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032743    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:22:56.032802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032759    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:22:56.032802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032786    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:56.032802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032793    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032795    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032826    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032787    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032817    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032872    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032877    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032877    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:22:56.032973 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.032893    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:22:56.033332 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.033308    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:22:56.048594 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048734    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.048772 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2qn\" (UniqueName: \"kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048915    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccjg\" (UniqueName: \"kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.048945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049141 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.049180    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjsw\" (UniqueName: \"kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.049271    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:22:56.549227451 +0000 UTC m=+2.338784217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxt2k\" (UniqueName: \"kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.049690 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049569    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049583    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049297    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049646    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049764    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049764    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.050337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049802    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049989    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.049997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051135 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050137    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxbq\" (UniqueName: \"kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050319    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050394    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgwh\" (UniqueName: \"kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjcd\" (UniqueName: \"kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050684    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.051887 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050952    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.050987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051009    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051103    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051128    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.052547 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051157    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051204    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.051238    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051310    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.051356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:22:56.55127485 +0000 UTC m=+2.340831611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051516    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051564    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.053159 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051628    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mzv\" (UniqueName: \"kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh77z\" (UniqueName: \"kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.051955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.053794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052192    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.052840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.053787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.053849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.054047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.054074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.054271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.054402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.054338    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.055861 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.055844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:56.106411 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.106371    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:56.106411 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.106410    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:56.106411 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.106426    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:56.106768 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.106493    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:56.60647458 +0000 UTC m=+2.396031338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:56.109817 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.109733    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxt2k\" (UniqueName: \"kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.109817 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.109786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2qn\" (UniqueName: \"kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.110111 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.110087    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccjg\" (UniqueName: \"kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.110196 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.110111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjsw\" (UniqueName: \"kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.110573 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.110546    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxbq\" (UniqueName: \"kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.110680 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.110576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgwh\" (UniqueName: \"kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.111075 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.111052    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjcd\" (UniqueName: \"kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.111130 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.111113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mzv\" (UniqueName: \"kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.115036 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.115012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh77z\" (UniqueName: \"kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:56.214407 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.214374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:22:56.239411 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.239379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:22:56.241722 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.241701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:22:56.270801 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.270753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nsn92"
Nov 01 22:22:56.270980 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.270831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:22:56.303846 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.303808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:22:56.308081 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.308054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:22:56.323443 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.323410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:22:56.328507 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.328477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:22:56.425827 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.425760    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.448736 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.448717    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:22:56.450135 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:56.450102    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d418d10f70ddaf0baab50de85c3c25b.slice/crio-843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2 WatchSource:0}: Error finding container 843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2: Status 404 returned error can't find the container with id 843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2
Nov 01 22:22:56.556869 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.556830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:56.557060 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.556888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:56.557060 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.556983    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:56.557060 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.556992    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:56.557060 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.557039    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:22:57.557024407 +0000 UTC m=+3.346581151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:56.557060 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.557054    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:22:57.557046436 +0000 UTC m=+3.346603181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:56.658142 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.658110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:56.658767 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.658233    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:56.658767 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.658248    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:56.658767 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.658257    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:56.658767 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:56.658307    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:57.658293955 +0000 UTC m=+3.447850700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:56.790344 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.790228    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"7f3192736cdd118e9e4731ee3fb56338","Type":"ContainerStarted","Data":"a58c23e3e29ff833f6fe08c0959b0f56fb9b3dbf9d2cd5c533c9166d900d8a58"}
Nov 01 22:22:56.791315 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.791288    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerStarted","Data":"843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2"}
Nov 01 22:22:56.964451 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.964416    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.978254 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.978210    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-04-20 11:28:00.294925852 +0000 UTC"
Nov 01 22:22:56.978254 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:56.978248    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12853h5m3.316681412s"
Nov 01 22:22:57.059701 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.059664    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:57.130472 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.130436    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cf3107_3278_46ee_8ebd_165aa8bba436.slice/crio-32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d WatchSource:0}: Error finding container 32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d: Status 404 returned error can't find the container with id 32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d
Nov 01 22:22:57.176768 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.176734    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96be3757_1e1a_40a2_95f8_81b9193ef313.slice/crio-4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561 WatchSource:0}: Error finding container 4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561: Status 404 returned error can't find the container with id 4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561
Nov 01 22:22:57.203459 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.203430    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3a1880_066c_4255_9da2_d6fad273bc48.slice/crio-d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7 WatchSource:0}: Error finding container d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7: Status 404 returned error can't find the container with id d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7
Nov 01 22:22:57.204305 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.204271    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76d2c9f_944d_40e7_a5be_23725a4575b2.slice/crio-766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f WatchSource:0}: Error finding container 766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f: Status 404 returned error can't find the container with id 766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f
Nov 01 22:22:57.223456 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.223422    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0a8880_fd76_4eba_a768_a5e9577e986c.slice/crio-f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514 WatchSource:0}: Error finding container f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514: Status 404 returned error can't find the container with id f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514
Nov 01 22:22:57.267350 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.267306    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee86ae0_eaac_4b60_bfc3_bbd995ef3cab.slice/crio-56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af WatchSource:0}: Error finding container 56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af: Status 404 returned error can't find the container with id 56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af
Nov 01 22:22:57.267978 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.267954    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba42d5e6_f6fb_49ea_876b_8719a5c9c6bc.slice/crio-d5cd1e63ebd06a3cc6595d6a43d9015fb936c81c00cfdf7298b4a646af6b4402 WatchSource:0}: Error finding container d5cd1e63ebd06a3cc6595d6a43d9015fb936c81c00cfdf7298b4a646af6b4402: Status 404 returned error can't find the container with id d5cd1e63ebd06a3cc6595d6a43d9015fb936c81c00cfdf7298b4a646af6b4402
Nov 01 22:22:57.292802 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.292771    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21a67f5_fb49_4af9_90b2_b68da8775bd6.slice/crio-9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443 WatchSource:0}: Error finding container 9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443: Status 404 returned error can't find the container with id 9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443
Nov 01 22:22:57.319064 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:22:57.319025    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919cce7a_c6b0_4724_9b16_6314caf4716b.slice/crio-56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238 WatchSource:0}: Error finding container 56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238: Status 404 returned error can't find the container with id 56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238
Nov 01 22:22:57.567812 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.567706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:57.567812 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.567770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:57.568052 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.567887    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:57.568052 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.567895    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:57.568052 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.567950    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.567934303 +0000 UTC m=+5.357491047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:57.568052 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.567966    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.567959296 +0000 UTC m=+5.357516041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:57.668366 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.668325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:57.668916 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.668502    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:57.668916 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.668527    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:57.668916 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.668543    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:57.668916 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.668624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.668608487 +0000 UTC m=+5.458165231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:57.786903 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.786863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:57.787099 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.786910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:57.787099 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.786910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:57.787099 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.787022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:22:57.787264 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.787109    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:22:57.787264 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:57.787192    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:22:57.795087 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.795046    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hxbq6" event={"ID":"a21a67f5-fb49-4af9-90b2-b68da8775bd6","Type":"ContainerStarted","Data":"9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443"}
Nov 01 22:22:57.796625 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.796591    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7rnx" event={"ID":"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc","Type":"ContainerStarted","Data":"d5cd1e63ebd06a3cc6595d6a43d9015fb936c81c00cfdf7298b4a646af6b4402"}
Nov 01 22:22:57.798703 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.798671    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af"}
Nov 01 22:22:57.799945 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.799911    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerStarted","Data":"f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514"}
Nov 01 22:22:57.801275 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.801239    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46rjc" event={"ID":"f76d2c9f-944d-40e7-a5be-23725a4575b2","Type":"ContainerStarted","Data":"766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f"}
Nov 01 22:22:57.802637 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.802609    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nsn92" event={"ID":"4e3a1880-066c-4255-9da2-d6fad273bc48","Type":"ContainerStarted","Data":"d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7"}
Nov 01 22:22:57.804596 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.804474    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x57zp" event={"ID":"919cce7a-c6b0-4724-9b16-6314caf4716b","Type":"ContainerStarted","Data":"56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238"}
Nov 01 22:22:57.805843 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.805816    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj" event={"ID":"96be3757-1e1a-40a2-95f8-81b9193ef313","Type":"ContainerStarted","Data":"4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561"}
Nov 01 22:22:57.807047 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.807005    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w67hg" event={"ID":"d2cf3107-3278-46ee-8ebd-165aa8bba436","Type":"ContainerStarted","Data":"32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d"}
Nov 01 22:22:57.979367 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.979252    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-08-10 22:26:31.513847876 +0000 UTC"
Nov 01 22:22:57.979367 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:57.979309    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15552h3m33.534541831s"
Nov 01 22:22:59.588194 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.588142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:59.588998 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.588221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:59.588998 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.588373    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.588998 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.588440    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:23:03.588420655 +0000 UTC m=+9.377977413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.588998 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.588508    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.588998 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.588540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:23:03.588531042 +0000 UTC m=+9.378087803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.688604 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.688527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:59.688853 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.688710    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:59.688853 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.688734    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:59.688853 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.688747    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.688853 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.688806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:03.688787958 +0000 UTC m=+9.478344709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.786793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.786933    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.787019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.787102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:22:59.787160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:22:59.787351 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:22:59.787239    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:01.786867 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.786831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:01.787397 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:01.786997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:01.787397 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.787138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:01.787397 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:01.787235    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:01.787397 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.787284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:01.787397 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:01.787355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:01.829122 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.829086    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d418d10f70ddaf0baab50de85c3c25b" containerID="7fb98ef48322557995f47f34f2626ea9da42fa2d3c7aa2964c817c9362441241" exitCode=0
Nov 01 22:23:01.829692 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.829660    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerDied","Data":"7fb98ef48322557995f47f34f2626ea9da42fa2d3c7aa2964c817c9362441241"}
Nov 01 22:23:01.842155 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.841787    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"7f3192736cdd118e9e4731ee3fb56338","Type":"ContainerStarted","Data":"8a6a3264f3c9022925015e645dc990ca766182b6c09fd44de6934db96c5d9086"}
Nov 01 22:23:01.848481 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.847832    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="27513e9031d8c149ae877dc9b17ac9e34ab9e5d525a534dea5ead1f7ab49ff44" exitCode=0
Nov 01 22:23:01.848481 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:01.847890    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"27513e9031d8c149ae877dc9b17ac9e34ab9e5d525a534dea5ead1f7ab49ff44"}
Nov 01 22:23:02.854820 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:02.854777    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hxbq6" event={"ID":"a21a67f5-fb49-4af9-90b2-b68da8775bd6","Type":"ContainerStarted","Data":"c323196cffae0b245e2ac8767d3186d5a3f4de87d48d4f4a905ab88deb295cbe"}
Nov 01 22:23:02.867291 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:02.867169    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerStarted","Data":"28bd7bed9f1cabb9cc11a5922c238af3847294ec2fd69908c93fd1cdaab88333"}
Nov 01 22:23:02.972899 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:02.972823    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal" podStartSLOduration=7.972804142 podStartE2EDuration="7.972804142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.191341263 +0000 UTC m=+7.980898034" watchObservedRunningTime="2026-11-01 22:23:02.972804142 +0000 UTC m=+8.762360913"
Nov 01 22:23:02.973305 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:02.973267    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hxbq6" podStartSLOduration=2.653647956 podStartE2EDuration="7.973246967s" totalImagesPullingTime="5.319599011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.971953702 +0000 UTC m=+8.761510472" watchObservedRunningTime="2026-11-01 22:23:02.973246967 +0000 UTC m=+8.762803736"
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.627737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.627791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.627891    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.627941    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.627927588 +0000 UTC m=+17.417484347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.628297    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:03.628398 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.628338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.628326705 +0000 UTC m=+17.417883464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:03.730864 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.728233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:03.730864 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.728405    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:03.730864 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.728437    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:03.730864 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.728450    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:03.730864 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.728521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:11.728496023 +0000 UTC m=+17.518052788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.786524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.786524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.786705    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:03.786750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.786837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:03.786952 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:03.786921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:04.795464 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:04.795431    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:23:04.795897 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:04.795526    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:23:04.880286 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:04.880239    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46rjc" event={"ID":"f76d2c9f-944d-40e7-a5be-23725a4575b2","Type":"ContainerStarted","Data":"6373d7e6c373b4945182f74e0ee76acd03e0c34098c814d2cd09bc41450d3e3c"}
Nov 01 22:23:04.885150 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:04.885116    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj" event={"ID":"96be3757-1e1a-40a2-95f8-81b9193ef313","Type":"ContainerStarted","Data":"4281bff95f08acaaee0ddff989681a0ca8221cf4c124a60738e58f9d41963224"}
Nov 01 22:23:04.913934 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:04.913515    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" podStartSLOduration=9.913495866 podStartE2EDuration="9.913495866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:03.09078996 +0000 UTC m=+8.880346728" watchObservedRunningTime="2026-11-01 22:23:04.913495866 +0000 UTC m=+10.703052635"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:05.786948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:05.787083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:05.787154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:05.787231    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:05.787282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:05.787746 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:05.787344    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:06.900062 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:06.900020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nsn92" event={"ID":"4e3a1880-066c-4255-9da2-d6fad273bc48","Type":"ContainerStarted","Data":"bec1a9e9990f6121614d9be19806799741b4c9fd12b6ed5fbf75df721ca212e1"}
Nov 01 22:23:06.982954 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:06.982262    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:23:06.982954 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:06.982428    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:23:06.984462 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:06.984362    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:23:07.030493 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.030401    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-46rjc" podStartSLOduration=5.393459334 podStartE2EDuration="12.030385128s" totalImagesPullingTime="6.636925794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:05.030634095 +0000 UTC m=+10.820190864" watchObservedRunningTime="2026-11-01 22:23:07.030385128 +0000 UTC m=+12.819941944"
Nov 01 22:23:07.098004 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.097950    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:23:07.159806 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.159753    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nsn92" podStartSLOduration=2.89956534 podStartE2EDuration="12.159736271s" totalImagesPullingTime="9.260170931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.051834184 +0000 UTC m=+12.841390949" watchObservedRunningTime="2026-11-01 22:23:07.159736271 +0000 UTC m=+12.949293036"
Nov 01 22:23:07.721113 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.720915    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:07.097969885Z","UUID":"0bf0e510-9c5a-426c-8101-6e36e806096d","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:23:07.722737 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.722687    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:23:07.722737 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.722725    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:23:07.786909 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.786758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:07.786909 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.786804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:07.786909 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.786758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:07.786909 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:07.786888    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:07.787219 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:07.787007    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:07.787219 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:07.787132    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:07.913420 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.913382    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x57zp" event={"ID":"919cce7a-c6b0-4724-9b16-6314caf4716b","Type":"ContainerStarted","Data":"cbf5720136da0f07ca729efdd301de4f1568c9866851f7373d2494d420b73084"}
Nov 01 22:23:07.915552 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.915504    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj" event={"ID":"96be3757-1e1a-40a2-95f8-81b9193ef313","Type":"ContainerStarted","Data":"7d632d3704cca9ca3086250e3ea1f9f51aba49b8fb83a2e8383a9914af60157e"}
Nov 01 22:23:07.917672 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.917646    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="64294f6adf671e2ff493db9b854fd2bd68b587a4a204ed6230897c22f7b82fd6" exitCode=0
Nov 01 22:23:07.917756 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:07.917743    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"64294f6adf671e2ff493db9b854fd2bd68b587a4a204ed6230897c22f7b82fd6"}
Nov 01 22:23:08.022032 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:08.021791    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x57zp" podStartSLOduration=2.384484418 podStartE2EDuration="12.021775982s" totalImagesPullingTime="9.637291564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:08.021139257 +0000 UTC m=+13.810696022" watchObservedRunningTime="2026-11-01 22:23:08.021775982 +0000 UTC m=+13.811332748"
Nov 01 22:23:08.921077 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:08.921038    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q7rnx" event={"ID":"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc","Type":"ContainerStarted","Data":"beb7204ec6045de4139b83aca373c9e35a54eb319fb4c088b56d725f9a05a205"}
Nov 01 22:23:08.923455 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:08.923429    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj" event={"ID":"96be3757-1e1a-40a2-95f8-81b9193ef313","Type":"ContainerStarted","Data":"369afc3df0fbce999889a35346686e4cf7a06922458a748b677e8b36b8305a7f"}
Nov 01 22:23:08.925283 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:08.925252    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w67hg" event={"ID":"d2cf3107-3278-46ee-8ebd-165aa8bba436","Type":"ContainerStarted","Data":"fcddc8ae0546e0bb1c73e705a4d9b4695419915db24fa62438c56fac41b748a1"}
Nov 01 22:23:09.031738 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.031622    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q7rnx" podStartSLOduration=2.573638937 podStartE2EDuration="14.031608048s" totalImagesPullingTime="11.457969111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.031309411 +0000 UTC m=+14.820866177" watchObservedRunningTime="2026-11-01 22:23:09.031608048 +0000 UTC m=+14.821164813"
Nov 01 22:23:09.141194 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.141147    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj" podStartSLOduration=3.073201819 podStartE2EDuration="14.141128072s" totalImagesPullingTime="11.067926253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.141012841 +0000 UTC m=+14.930569606" watchObservedRunningTime="2026-11-01 22:23:09.141128072 +0000 UTC m=+14.930684838"
Nov 01 22:23:09.251993 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.251933    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w67hg" podStartSLOduration=4.603112784 podStartE2EDuration="14.251913762s" totalImagesPullingTime="9.648800978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.251704059 +0000 UTC m=+15.041260824" watchObservedRunningTime="2026-11-01 22:23:09.251913762 +0000 UTC m=+15.041470528"
Nov 01 22:23:09.786637 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.786602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:09.786824 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.786602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:09.786824 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:09.786734    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:09.786824 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.786613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:09.786976 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:09.786841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:09.786976 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:09.786899    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:09.929393 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.929359    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="a18d38ca0b6fbd68ca203ff484a3c83712810adb977c5f91c24e9b9e50999993" exitCode=0
Nov 01 22:23:09.929907 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:09.929456    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"a18d38ca0b6fbd68ca203ff484a3c83712810adb977c5f91c24e9b9e50999993"}
Nov 01 22:23:11.694672 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.694631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:11.695275 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.694702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:11.695275 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.694789    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:11.695275 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.694789    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:11.695275 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.694854    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.694836708 +0000 UTC m=+33.484393453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:11.695275 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.694872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.694862731 +0000 UTC m=+33.484419473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:11.786400 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.786366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:11.786607 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.786412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:11.786607 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.786366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:11.786607 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.786521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:11.786758 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.786639    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:11.786758 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.786728    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:11.795631 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.795574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:11.795788 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.795739    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:11.795788 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.795756    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:11.795788 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.795768    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sk465 for pod openshift-network-diagnostics/network-check-target-7flsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:11.795950 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:11.795820    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465 podName:95d3c296-d0fc-4987-bc94-0ff8ab739120 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.79580749 +0000 UTC m=+33.585364254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sk465" (UniqueName: "kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465") pod "network-check-target-7flsn" (UID: "95d3c296-d0fc-4987-bc94-0ff8ab739120") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:11.937146 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.937114    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="b53f1e9caacb3631ee1f56d3bd86627793fe20911c19be1bfbbba528591c8809" exitCode=0
Nov 01 22:23:11.937302 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:11.937172    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"b53f1e9caacb3631ee1f56d3bd86627793fe20911c19be1bfbbba528591c8809"}
Nov 01 22:23:13.787005 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:13.786773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:13.787426 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:13.787118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:13.787599 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:13.787570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:13.787714 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:13.787684    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:13.787848 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:13.787749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:13.787848 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:13.787814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:14.949825 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.949624    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"ba8c4088a0fd6fa864b9cbb3f958357698f8731c079496a66eba18e483b319fd"}
Nov 01 22:23:14.950389 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.949917    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"5f2915cd8925e4ebf8de31e0b245c82ddf423add347f35f1e60483d744f51e95"}
Nov 01 22:23:14.950389 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.949962    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"7abe7743fa550e5ae628588b102de3945d8e86d22bb01cf082b208faf710417b"}
Nov 01 22:23:14.950389 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.950005    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"65d0cd9ebf33c8252cb9cd4608e772d96bd30330e9a4e85106835e11f49c6d63"}
Nov 01 22:23:14.950389 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.950039    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"23882ff19f03ecef2b1ba75649a618bd5e0ff0ab1a95da2c6cc460941ad7d7f4"}
Nov 01 22:23:14.950389 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.950072    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"659946c60073baf826b764e0d3ecc1a9530c889715fdf3e2ce57eb3f5b710033"}
Nov 01 22:23:14.952265 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.952241    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="8bd962f49a1714e686f363eb15bd37903deea8ce61e93e76ed2c493120a1cf4a" exitCode=0
Nov 01 22:23:14.952361 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:14.952270    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"8bd962f49a1714e686f363eb15bd37903deea8ce61e93e76ed2c493120a1cf4a"}
Nov 01 22:23:15.786482 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:15.786451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:15.786482 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:15.786470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:15.786714 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:15.786582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:15.786714 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:15.786628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:15.786778 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:15.786718    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:15.786810 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:15.786779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:15.957632 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:15.957604    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="44daf30590960209ea984969b2c655cb1bfad7898ea8b2b65d9a14f41af12397" exitCode=0
Nov 01 22:23:15.958067 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:15.957648    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerDied","Data":"44daf30590960209ea984969b2c655cb1bfad7898ea8b2b65d9a14f41af12397"}
Nov 01 22:23:16.962746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:16.962659    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"7c6020870ab7f56cae969c76a50b3926dc163151334fd392f15da08bc3cdba70"}
Nov 01 22:23:16.965815 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:16.965792    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2prlz" event={"ID":"6f0a8880-fd76-4eba-a768-a5e9577e986c","Type":"ContainerStarted","Data":"6b745a407b8c088c4e946cdee8f797e6ccbff960891a7b0a3e2fc4d2a19ef55f"}
Nov 01 22:23:17.082105 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:17.082049    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2prlz" podStartSLOduration=8.510461621 podStartE2EDuration="22.082035751s" totalImagesPullingTime="13.57157413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:17.081527679 +0000 UTC m=+22.871084445" watchObservedRunningTime="2026-11-01 22:23:17.082035751 +0000 UTC m=+22.871592515"
Nov 01 22:23:17.786425 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:17.786380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:17.786623 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:17.786403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:17.786623 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:17.786512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:17.786623 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:17.786567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:17.786752 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:17.786661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:17.786752 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:17.786729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:18.973931 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:18.973844    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" event={"ID":"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab","Type":"ContainerStarted","Data":"590ac7cc7028ff40b9e37575d2373554beb6c63240df9d1fb26498a5a1f507b7"}
Nov 01 22:23:18.974631 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:18.974071    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:18.990133 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:18.990106    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:19.249859 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.249618    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs" podStartSLOduration=7.33484513 podStartE2EDuration="24.249604755s" totalImagesPullingTime="16.914759625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:19.249371813 +0000 UTC m=+25.038928578" watchObservedRunningTime="2026-11-01 22:23:19.249604755 +0000 UTC m=+25.039161520"
Nov 01 22:23:19.786214 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.786178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:19.786402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.786255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:19.786402 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.786285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:19.786402 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:19.786371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:19.786521 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:19.786409    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:19.786521 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:19.786486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:19.977260 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.977225    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:19.977670 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.977314    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:19.994651 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:19.994620    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:21.476129 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.476019    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7flsn"]
Nov 01 22:23:21.476593 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.476141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:21.476593 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:21.476248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:21.478972 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.478940    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7thn"]
Nov 01 22:23:21.479112 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.479037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:21.479159 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:21.479115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:21.523637 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.523603    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdh5p"]
Nov 01 22:23:21.523795 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:21.523712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:21.523840 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:21.523813    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:22.790436 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:22.790408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:22.790954 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:22.790509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:23.786670 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:23.786589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:23.786802 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:23.786696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:23.786802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:23.786744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:23.786887 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:23.786845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:24.787832 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:24.787799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:24.788196 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:24.787899    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdh5p" podUID="9fa7dbfa-8562-4396-9657-18862faf25dc"
Nov 01 22:23:25.786192 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:25.786148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:25.786192 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:25.786177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:25.786417 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:25.786260    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7flsn" podUID="95d3c296-d0fc-4987-bc94-0ff8ab739120"
Nov 01 22:23:25.786455 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:25.786417    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7thn" podUID="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb"
Nov 01 22:23:26.505185 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.505104    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeReady"
Nov 01 22:23:26.505578 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.505261    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 22:23:26.613270 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.613234    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sddpb"]
Nov 01 22:23:26.633656 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.633623    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tv2zn"]
Nov 01 22:23:26.633773 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.633741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.648807 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.648780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.651330 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.651303    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sddpb"]
Nov 01 22:23:26.655238 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.655210    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tv2zn"]
Nov 01 22:23:26.683802 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.683769    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:23:26.683986 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.683968    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:26.687380 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.687357    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:23:26.700639 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.700609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br79m\" (UniqueName: \"kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.700814 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.700648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.700814 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.700751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.700814 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.700776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.784735 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.784661    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:23:26.784735 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.784678    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:23:26.786972 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.786953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:26.787696 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.787679    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:23:26.787779 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.787708    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:23:26.801784 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.801764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrmt\" (UniqueName: \"kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.801883 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.801798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br79m\" (UniqueName: \"kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.801883 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.801826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.801883 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.801877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.802087 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.802065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.802146 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.802104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.802255 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.802237    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.802313 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:26.802246    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:26.802367 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:26.802329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.302308855 +0000 UTC m=+33.091865613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:26.802482 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.802464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.859813 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.859785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br79m\" (UniqueName: \"kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:26.885377 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.885344    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:23:26.885556 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.885398    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:26.902734 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.902706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrmt\" (UniqueName: \"kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.902815 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.902746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.905337 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.905308    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.962962 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.962933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrmt\" (UniqueName: \"kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:26.979916 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:26.979882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:23:27.306876 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.306836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:27.307054 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.306998    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.307095 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.307066    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.307050181 +0000 UTC m=+34.096606925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.431993 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.430751    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tv2zn"]
Nov 01 22:23:27.437553 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:27.437508    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c842552_5f6f_4868_8b12_5c97e1e35eb8.slice/crio-427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c WatchSource:0}: Error finding container 427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c: Status 404 returned error can't find the container with id 427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c
Nov 01 22:23:27.710610 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.710485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:27.710610 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.710601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:27.711076 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.710644    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:27.711076 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.710716    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret podName:af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb nodeName:}" failed. No retries permitted until 2026-11-01 22:23:59.710699631 +0000 UTC m=+65.500256377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret") pod "global-pull-secret-syncer-n7thn" (UID: "af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:27.711076 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.710734    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:27.711076 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:27.710827    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs podName:9fa7dbfa-8562-4396-9657-18862faf25dc nodeName:}" failed. No retries permitted until 2026-11-01 22:23:59.710810272 +0000 UTC m=+65.500367015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs") pod "network-metrics-daemon-sdh5p" (UID: "9fa7dbfa-8562-4396-9657-18862faf25dc") : secret "metrics-daemon-secret" not found
Nov 01 22:23:27.786797 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.786759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:27.786797 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.786785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:27.811267 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.811230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:27.870316 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.870280    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:23:27.870484 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.870446    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:23:27.870561 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.870499    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:27.870561 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.870554    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:23:27.870632 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.870582    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:27.874084 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.874062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:27.996169 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:27.996086    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tv2zn" event={"ID":"0c842552-5f6f-4868-8b12-5c97e1e35eb8","Type":"ContainerStarted","Data":"427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c"}
Nov 01 22:23:28.128150 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:28.128109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:28.314513 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:28.314474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:28.314707 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:28.314658    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:28.314750 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:28.314726    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.314708822 +0000 UTC m=+36.104265582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:28.524586 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:28.524531    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7flsn"]
Nov 01 22:23:28.531006 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:28.530981    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d3c296_d0fc_4987_bc94_0ff8ab739120.slice/crio-62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103 WatchSource:0}: Error finding container 62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103: Status 404 returned error can't find the container with id 62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103
Nov 01 22:23:29.000194 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:29.000146    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7flsn" event={"ID":"95d3c296-d0fc-4987-bc94-0ff8ab739120","Type":"ContainerStarted","Data":"62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103"}
Nov 01 22:23:30.330243 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:30.330200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:30.330678 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:30.330371    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:30.330678 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:30.330475    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.330460018 +0000 UTC m=+40.120016760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:32.010161 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:32.010112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tv2zn" event={"ID":"0c842552-5f6f-4868-8b12-5c97e1e35eb8","Type":"ContainerStarted","Data":"f9ae80e1f3d6bb2a499938e4be4026f3651b8728ecc27801f49c3cc543e39eba"}
Nov 01 22:23:32.120836 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:32.120770    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tv2zn" podStartSLOduration=2.064307876 podStartE2EDuration="6.120749706s" totalImagesPullingTime="4.05644183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:32.119520422 +0000 UTC m=+37.909077187" watchObservedRunningTime="2026-11-01 22:23:32.120749706 +0000 UTC m=+37.910306470"
Nov 01 22:23:33.016244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.016151    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7flsn" event={"ID":"95d3c296-d0fc-4987-bc94-0ff8ab739120","Type":"ContainerStarted","Data":"741f461129d09b6b15746385edabd26665d56d2787f8fdf5ae727f9b0a73f3a3"}
Nov 01 22:23:33.016244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.016233    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:23:33.125662 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.125605    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7flsn" podStartSLOduration=33.921870448 podStartE2EDuration="38.12559117s" totalImagesPullingTime="4.203720722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:33.124697743 +0000 UTC m=+38.914254523" watchObservedRunningTime="2026-11-01 22:23:33.12559117 +0000 UTC m=+38.915147934"
Nov 01 22:23:33.492102 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.492067    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5bshs"]
Nov 01 22:23:33.494464 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.494445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.544835 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.544806    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:23:33.544835 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.544827    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:23:33.545051 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.544839    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:23:33.545051 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.544860    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:23:33.592525 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.592489    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:23:33.593338 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.593319    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bshs"]
Nov 01 22:23:33.661424 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.661384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.661424 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.661431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.661712 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.661461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.661712 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.661520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.661712 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.661588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5xp\" (UniqueName: \"kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.762819 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.762708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.762819 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.762776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.762819 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.762807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5xp\" (UniqueName: \"kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:33.762842    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.762871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.762902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:33.762915    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.26290024 +0000 UTC m=+40.052457000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.763010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:33.763019    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.763099 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:33.763072    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.26305937 +0000 UTC m=+40.052616113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.763371 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.763218    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:33.819801 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:33.819767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5xp\" (UniqueName: \"kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:34.267052 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.267010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:34.267574 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.267108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:34.267574 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.267189    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.267574 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.267223    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.267574 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.267272    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.267252791 +0000 UTC m=+41.056809550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.267574 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.267294    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.267280891 +0000 UTC m=+41.056837635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.368424 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.368384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:34.368628 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.368586    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:34.368759 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:34.368670    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:42.368649166 +0000 UTC m=+48.158205910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:34.667633 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.667591    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"]
Nov 01 22:23:34.669513 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.669490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:23:34.721782 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.721756    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jjz5d\""
Nov 01 22:23:34.724686 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.724661    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:34.724839 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.724749    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 22:23:34.725980 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.725957    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"]
Nov 01 22:23:34.772369 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.772330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7wv\" (UniqueName: \"kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv\") pod \"migrator-66df5d496b-7qhgf\" (UID: \"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:23:34.872983 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.872937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7wv\" (UniqueName: \"kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv\") pod \"migrator-66df5d496b-7qhgf\" (UID: \"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:23:34.927520 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.927451    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7wv\" (UniqueName: \"kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv\") pod \"migrator-66df5d496b-7qhgf\" (UID: \"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:23:34.979527 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:34.979492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:23:35.275873 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:35.275788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:35.275873 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:35.275862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:35.276266 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:35.275925    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.276266 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:35.275958    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:35.276266 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:35.275992    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.275976002 +0000 UTC m=+43.065532765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.276266 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:35.276007    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.276000597 +0000 UTC m=+43.065557340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:35.352068 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:35.352035    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"]
Nov 01 22:23:35.358823 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:35.358796    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd937c5d0_bdd4_4e16_8f01_51750bdc2c7a.slice/crio-da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd WatchSource:0}: Error finding container da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd: Status 404 returned error can't find the container with id da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd
Nov 01 22:23:36.025362 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:36.025329    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf" event={"ID":"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a","Type":"ContainerStarted","Data":"da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd"}
Nov 01 22:23:37.293030 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:37.292992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:37.293469 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:37.293043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:37.293469 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:37.293121    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:37.293469 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:37.293169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.293157279 +0000 UTC m=+47.082714035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:37.293469 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:37.293172    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:37.293469 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:37.293239    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.293221454 +0000 UTC m=+47.082778210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:38.031504 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:38.031458    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf" event={"ID":"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a","Type":"ContainerStarted","Data":"84f7a51882a9fedce8b9c9b68bdccb789c473f8ed761c627658463712d8a2e00"}
Nov 01 22:23:38.145021 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:38.144965    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf" podStartSLOduration=2.327777824 podStartE2EDuration="4.144950815s" totalImagesPullingTime="1.817172991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:38.144134242 +0000 UTC m=+43.933691007" watchObservedRunningTime="2026-11-01 22:23:38.144950815 +0000 UTC m=+43.934507579"
Nov 01 22:23:39.601638 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:39.601595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hxbq6_a21a67f5-fb49-4af9-90b2-b68da8775bd6/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:41.323218 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:41.323157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:41.323766 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:41.323269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:41.323766 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:41.323390    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.323766 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:41.323467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls podName:b8a1f158-f781-40f4-a9c4-899b928dbf8a nodeName:}" failed. No retries permitted until 2026-11-01 22:23:49.323447445 +0000 UTC m=+55.113004209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bshs" (UID: "b8a1f158-f781-40f4-a9c4-899b928dbf8a") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.323766 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:41.323735    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:42.372218 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:42.372194    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x57zp_919cce7a-c6b0-4724-9b16-6314caf4716b/dns-node-resolver/0.log"
Nov 01 22:23:42.431311 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:42.431272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:42.431467 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:42.431417    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:42.431504 ip-10-0-138-155 kubenswrapper[2577]: E1101 22:23:42.431478    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls podName:907b53dd-5fe8-4df2-b5eb-d0e487157df3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:58.43145985 +0000 UTC m=+64.221016594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls") pod "dns-default-sddpb" (UID: "907b53dd-5fe8-4df2-b5eb-d0e487157df3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:42.972475 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:42.972454    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46rjc_f76d2c9f-944d-40e7-a5be-23725a4575b2/node-ca/0.log"
Nov 01 22:23:44.172349 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:44.172324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tv2zn_0c842552-5f6f-4868-8b12-5c97e1e35eb8/serve-healthcheck-canary/0.log"
Nov 01 22:23:44.372963 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:44.372932    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-7qhgf_d937c5d0-bdd4-4e16-8f01-51750bdc2c7a/migrator/0.log"
Nov 01 22:23:49.382209 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:49.382166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:49.384730 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:49.384705    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:49.406746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:49.406711    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:23:49.772468 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:49.772376    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bshs"]
Nov 01 22:23:49.779690 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:49.779641    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a1f158_f781_40f4_a9c4_899b928dbf8a.slice/crio-e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d WatchSource:0}: Error finding container e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d: Status 404 returned error can't find the container with id e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d
Nov 01 22:23:50.060085 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:50.060046    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bshs" event={"ID":"b8a1f158-f781-40f4-a9c4-899b928dbf8a","Type":"ContainerStarted","Data":"8e7f89e28484e61d7b8d087cfc9ce4971334eff43da771dd7a1349cc42257000"}
Nov 01 22:23:50.060241 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:50.060105    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bshs" event={"ID":"b8a1f158-f781-40f4-a9c4-899b928dbf8a","Type":"ContainerStarted","Data":"e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d"}
Nov 01 22:23:51.993244 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:51.993220    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:23:52.067296 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:52.067261    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bshs" event={"ID":"b8a1f158-f781-40f4-a9c4-899b928dbf8a","Type":"ContainerStarted","Data":"b3c013f11f6bc820a0a6679b140ba3327d66562fa0c0d705cdf4fe853f814f53"}
Nov 01 22:23:54.074356 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.074320    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bshs" event={"ID":"b8a1f158-f781-40f4-a9c4-899b928dbf8a","Type":"ContainerStarted","Data":"5862d1e80e67bcdf2154f02c9b25bcd19563426a6171dbc0bdf9efd02732be17"}
Nov 01 22:23:54.185878 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.185826    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5bshs" podStartSLOduration=18.036293631 podStartE2EDuration="21.185810778s" totalImagesPullingTime="3.149517147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:23:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:54.1852388 +0000 UTC m=+59.974795566" watchObservedRunningTime="2026-11-01 22:23:54.185810778 +0000 UTC m=+59.975367544"
Nov 01 22:23:54.907199 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.907161    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ddd6b4776-stdwm"]
Nov 01 22:23:54.911276 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.911260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919371 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919515 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919515 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919515 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919515 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9slz\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919702 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919702 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.919766 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.919714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:54.954092 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.954063    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ddd6b4776-stdwm"]
Nov 01 22:23:54.988491 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:54.988460    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 22:23:55.000755 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.000712    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 22:23:55.000755 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.000713    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bn2bj\""
Nov 01 22:23:55.020744 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.020866 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.020909 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.020955 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.020955 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.021090 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.021090 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.021090 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.020990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9slz\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.021253 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.021213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.021662 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.021642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.023581 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.023533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.023656 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.023636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.048820 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.048788    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 22:23:55.053413 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.053382    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.075247 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.075213    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 22:23:55.082883 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.082859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.086953 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.086928    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.087066 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.086966    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9slz\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.221935 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.221839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:55.706468 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:55.706436    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ddd6b4776-stdwm"]
Nov 01 22:23:55.735615 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:55.735582    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22260f24_c7ba_4462_ab58_c49f4bbbf1b1.slice/crio-260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72 WatchSource:0}: Error finding container 260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72: Status 404 returned error can't find the container with id 260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72
Nov 01 22:23:56.082512 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:56.082473    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm" event={"ID":"22260f24-c7ba-4462-ab58-c49f4bbbf1b1","Type":"ContainerStarted","Data":"b2eb6fa0bd6a0c236550f54f2c490da8eaf79b5b349077bcf49d805a89419e89"}
Nov 01 22:23:56.083047 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:56.082533    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm" event={"ID":"22260f24-c7ba-4462-ab58-c49f4bbbf1b1","Type":"ContainerStarted","Data":"260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72"}
Nov 01 22:23:56.083047 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:56.082592    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:23:56.199696 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:56.199643    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm" podStartSLOduration=2.199630123 podStartE2EDuration="2.199630123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:56.199124009 +0000 UTC m=+61.988680853" watchObservedRunningTime="2026-11-01 22:23:56.199630123 +0000 UTC m=+61.989186887"
Nov 01 22:23:58.449746 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.449712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:58.452180 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.452151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:58.516219 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.516189    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:58.517213 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.517194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sddpb"
Nov 01 22:23:58.846021 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.845989    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"]
Nov 01 22:23:58.849251 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.849235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.853083 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.853053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2qh\" (UniqueName: \"kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.853209 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.853118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.853209 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.853185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.853284 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.853227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.885093 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.885060    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sddpb"]
Nov 01 22:23:58.930220 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:58.930190    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907b53dd_5fe8_4df2_b5eb_d0e487157df3.slice/crio-8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707 WatchSource:0}: Error finding container 8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707: Status 404 returned error can't find the container with id 8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707
Nov 01 22:23:58.931521 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.931495    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 22:23:58.932214 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.931790    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 22:23:58.932214 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.931956    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 22:23:58.932214 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.931968    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dqcqf\""
Nov 01 22:23:58.932697 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.932677    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"]
Nov 01 22:23:58.949470 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.949438    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 22:23:58.949470 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.949449    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 22:23:58.953833 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.953808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2qh\" (UniqueName: \"kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.953948 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.953859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.953948 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.953881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.953948 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.953918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.954689 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.954630    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.956467 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.956444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:58.956596 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:58.956577    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:59.009344 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.009315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2qh\" (UniqueName: \"kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:59.093992 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.093958    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddpb" event={"ID":"907b53dd-5fe8-4df2-b5eb-d0e487157df3","Type":"ContainerStarted","Data":"8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707"}
Nov 01 22:23:59.162796 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.162696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:23:59.531585 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.531477    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"]
Nov 01 22:23:59.537888 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:23:59.537860    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc780c544_2eef_4d4c_8b46_b3f079ef2e72.slice/crio-33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941 WatchSource:0}: Error finding container 33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941: Status 404 returned error can't find the container with id 33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941
Nov 01 22:23:59.760218 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.760149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:59.760406 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.760250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:59.762857 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.762832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:59.809224 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.809197    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:59.813780 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.813760    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:23:59.869683 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.869652    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:59.871662 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.871643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:23:59.995875 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.995844    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:59.997008 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:23:59.996990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:00.099492 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:00.099399    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl" event={"ID":"c780c544-2eef-4d4c-8b46-b3f079ef2e72","Type":"ContainerStarted","Data":"33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941"}
Nov 01 22:24:00.241727 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:00.241688    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdh5p"]
Nov 01 22:24:00.365607 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:00.365507    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7thn"]
Nov 01 22:24:00.372259 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:24:00.372226    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf321a6a_4dd2_4105_9c8a_a5dbb8e6e3fb.slice/crio-fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e WatchSource:0}: Error finding container fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e: Status 404 returned error can't find the container with id fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e
Nov 01 22:24:01.104522 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:01.104485    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n7thn" event={"ID":"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb","Type":"ContainerStarted","Data":"fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e"}
Nov 01 22:24:01.106042 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:01.106013    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdh5p" event={"ID":"9fa7dbfa-8562-4396-9657-18862faf25dc","Type":"ContainerStarted","Data":"1cf76dc49297405baa7aaa49c78e0db0bdaae87769822760b5ea1d9c4405ce79"}
Nov 01 22:24:02.112210 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.112172    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl" event={"ID":"c780c544-2eef-4d4c-8b46-b3f079ef2e72","Type":"ContainerStarted","Data":"f5cae1ddf33d393e1c0c770c097b59c973d22fce813b8aa22b69ea5ec7faaf85"}
Nov 01 22:24:02.112652 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.112232    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl" event={"ID":"c780c544-2eef-4d4c-8b46-b3f079ef2e72","Type":"ContainerStarted","Data":"7cb5951f1d7cb26aff15c0b6a3842e6dd40507d992b1d83f0f9cbd3a875e82cc"}
Nov 01 22:24:02.114748 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.114715    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdh5p" event={"ID":"9fa7dbfa-8562-4396-9657-18862faf25dc","Type":"ContainerStarted","Data":"2d5ae8f9b60a6731d10a7483a06e4dcb14a4cd9ac06bf8255887b683f08f0c73"}
Nov 01 22:24:02.114900 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.114767    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdh5p" event={"ID":"9fa7dbfa-8562-4396-9657-18862faf25dc","Type":"ContainerStarted","Data":"8066b70e5d95ed27151c9d577a9517ab63c0018921c39876e6ec698b40ca1666"}
Nov 01 22:24:02.116960 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.116932    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddpb" event={"ID":"907b53dd-5fe8-4df2-b5eb-d0e487157df3","Type":"ContainerStarted","Data":"584fad7da4f3c9253d263a6822ecfbc0caca6228f64454017694b6141663cdd1"}
Nov 01 22:24:02.117081 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.116992    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:02.117081 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.117017    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sddpb" event={"ID":"907b53dd-5fe8-4df2-b5eb-d0e487157df3","Type":"ContainerStarted","Data":"5a06d736146c3fa8397340a22d65eece82cff3219a4e8a4ca1f0d4e79aa57254"}
Nov 01 22:24:02.241931 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.241829    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl" podStartSLOduration=2.215788149 podStartE2EDuration="4.241815443s" totalImagesPullingTime="2.026027294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:02.240800865 +0000 UTC m=+68.030357632" watchObservedRunningTime="2026-11-01 22:24:02.241815443 +0000 UTC m=+68.031372207"
Nov 01 22:24:02.352295 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.352235    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sdh5p" podStartSLOduration=65.718908357 podStartE2EDuration="1m7.352216452s" totalImagesPullingTime="1.633308095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:02.351619795 +0000 UTC m=+68.141176553" watchObservedRunningTime="2026-11-01 22:24:02.352216452 +0000 UTC m=+68.141773219"
Nov 01 22:24:02.461961 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:02.461909    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-sddpb" podStartSLOduration=34.205497559 podStartE2EDuration="36.461891751s" totalImagesPullingTime="2.256394192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:02.460874612 +0000 UTC m=+68.250431376" watchObservedRunningTime="2026-11-01 22:24:02.461891751 +0000 UTC m=+68.251448518"
Nov 01 22:24:04.021550 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:04.021505    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:24:05.547086 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.547049    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8d6wj"]
Nov 01 22:24:05.549600 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.549567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.601094 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.601053    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 22:24:05.604815 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.604778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.604815 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.604831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605048 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.604903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605048 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.604949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605048 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.605045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605177 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.605069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605177 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.605098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605271 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.605173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.605345 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.605264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlf8f\" (UniqueName: \"kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.648059 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.647987    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 22:24:05.648059 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.647987    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 22:24:05.648309 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.648071    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8bwdj\""
Nov 01 22:24:05.706636 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706636 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf8f\" (UniqueName: \"kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706731    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.706884 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707157 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707157 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.706940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707157 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.707051    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707273 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.707190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707357 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.707335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707423 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.707342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.707794 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.707771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.709253 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.709224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.709253 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.709246    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.761886 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.761850    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf8f\" (UniqueName: \"kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.864985 ip-10-0-138-155 kubenswrapper[2577]: I1101 22:24:05.864940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:05.878027 ip-10-0-138-155 kubenswrapper[2577]: W1101 22:24:05.877994    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac262171_cc91_4552_ad95_7af735ac5450.slice/crio-16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417 WatchSource:0}: Error finding container 16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417: Status 404 returned error can't find the container with id 16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417
Nov 01 22:24:05.986200 ip-10-0-138-155 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 22:24:05.998984 ip-10-0-138-155 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 22:24:05.999183 ip-10-0-138-155 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 22:24:05.999236 ip-10-0-138-155 systemd[1]: kubelet.service: Consumed 3.601s CPU time.
Nov 01 22:24:06.011292 ip-10-0-138-155 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:24:06.076525 ip-10-0-138-155 kubenswrapper[5031]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:24:06.076525 ip-10-0-138-155 kubenswrapper[5031]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:24:06.076525 ip-10-0-138-155 kubenswrapper[5031]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:24:06.076525 ip-10-0-138-155 kubenswrapper[5031]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:24:06.078607 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078491    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.078607 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078606    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.078607 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078611    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078614    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078617    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078620    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078623    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078625    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078628    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078632    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078635    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078637    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078640    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078642    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078645    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078647    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078650    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078652    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078655    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078658    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078661    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078663    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.078716 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078666    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078668    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078671    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078673    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078677    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078680    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078682    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078686    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078688    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078691    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078693    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078696    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078699    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078702    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078704    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078707    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078710    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078712    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078715    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078717    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.079217 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078720    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078723    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078725    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078729    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078732    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078734    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078737    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078741    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078744    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078746    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078749    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078751    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078754    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078757    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078760    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078762    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078765    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078768    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078771    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078774    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.079856 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078777    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078780    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078782    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078785    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078787    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078790    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078793    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078795    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078798    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078801    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078803    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078808    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078812    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078815    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078818    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078820    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078823    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078826    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078828    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.080359 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078831    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078833    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078837    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078842    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078846    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078849    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078851    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078854    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078857    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078860    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078863    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078866    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078870    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078872    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078875    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078878    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078881    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078883    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078886    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.080858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078888    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078891    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078894    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078897    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078899    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.078902    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079755    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079764    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079768    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079771    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079773    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079777    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079780    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079783    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079785    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079788    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079791    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079793    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079796    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.081421 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079798    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079801    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079803    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079807    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079810    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079813    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079816    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079819    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079821    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079824    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079826    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079829    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079831    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079834    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079837    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079840    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079842    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079844    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079847    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079850    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.081927 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079852    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079855    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079858    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079861    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079864    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079867    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079870    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079872    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079875    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079878    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079882    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079886    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079889    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079892    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079895    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079898    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079900    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079903    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079905    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.082455 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079908    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079911    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079913    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079916    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079918    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079921    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079923    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079926    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079928    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079931    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079933    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079935    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079938    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079941    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079943    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079946    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079950    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079953    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079955    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079968    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.083070 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079971    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079974    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079977    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079979    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079982    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079984    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079987    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079989    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079992    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079995    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.079997    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080000    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080002    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080005    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080007    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080010    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080013    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080015    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080017    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.083608 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080020    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080022    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080025    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080027    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080030    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080032    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080040    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080044    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080048    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080051    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080054    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080057    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080060    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080065    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080068    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080601    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080607    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080609    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080613    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080616    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.084090 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080619    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080622    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080625    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080627    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080630    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080633    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080636    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080638    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080641    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080643    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080646    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080649    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080651    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080654    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080657    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080659    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080662    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080664    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.084603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080671    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080675    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080678    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080690    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080693    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080695    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080698    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080706    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080709    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080711    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080714    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080717    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080720    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080722    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080725    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080728    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080731    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080733    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080736    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080738    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.085249 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080741    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080743    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080746    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080748    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080751    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080754    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080756    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080758    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080761    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080763    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080766    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080768    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080774    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080778    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080781    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080784    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080787    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080789    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080792    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.086129 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080795    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080798    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080801    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080803    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080806    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080808    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080811    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080813    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080816    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080819    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080821    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080824    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080827    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080829    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080832    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080834    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080837    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080839    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080842    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080844    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.086900 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080847    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080849    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080852    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080854    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080857    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080861    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080863    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080866    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080869    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080872    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080874    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080877    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080880    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080883    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080886    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080889    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080891    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080894    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080896    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080899    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.087437 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080902    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080904    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080907    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.080909    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081016    5031 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081028    5031 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081038    5031 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081045    5031 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081052    5031 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081057    5031 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081063    5031 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081067    5031 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081071    5031 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081074    5031 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081077    5031 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081080    5031 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081083    5031 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081088    5031 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081091    5031 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081094    5031 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081097    5031 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081100    5031 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081104    5031 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:24:06.087973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081107    5031 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081110    5031 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081113    5031 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081116    5031 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081122    5031 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081125    5031 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081128    5031 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081131    5031 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081134    5031 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081137    5031 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081140    5031 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081144    5031 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081146    5031 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081151    5031 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081154    5031 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081157    5031 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081160    5031 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081163    5031 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081165    5031 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081172    5031 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081175    5031 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081177    5031 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081181    5031 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081184    5031 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:24:06.088642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081189    5031 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081192    5031 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081197    5031 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081200    5031 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081203    5031 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081206    5031 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081209    5031 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081212    5031 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081216    5031 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081219    5031 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081222    5031 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081226    5031 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081229    5031 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081236    5031 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081239    5031 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081242    5031 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081245    5031 flags.go:64] FLAG: --help="false"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081248    5031 flags.go:64] FLAG: --hostname-override="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081251    5031 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081254    5031 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081257    5031 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081260    5031 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081264    5031 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:24:06.089296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081267    5031 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081269    5031 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081272    5031 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081275    5031 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081278    5031 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081281    5031 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081283    5031 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081286    5031 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081290    5031 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081293    5031 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081296    5031 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081300    5031 flags.go:64] FLAG: --lock-file=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081303    5031 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081305    5031 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081310    5031 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081315    5031 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081318    5031 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081323    5031 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081326    5031 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081329    5031 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081332    5031 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081335    5031 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081338    5031 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081350    5031 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081353    5031 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:24:06.089949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081357    5031 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081360    5031 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081363    5031 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081366    5031 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081369    5031 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081372    5031 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081375    5031 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081378    5031 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081396    5031 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081399    5031 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081402    5031 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081405    5031 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081408    5031 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081411    5031 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081414    5031 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081417    5031 flags.go:64] FLAG: --port="10250"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081420    5031 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081423    5031 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08cba2557fe8b4eb1"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081427    5031 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081430    5031 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081433    5031 flags.go:64] FLAG: --register-node="true"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081436    5031 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081440    5031 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081443    5031 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081448    5031 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081451    5031 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:24:06.090569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081455    5031 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081458    5031 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081461    5031 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081463    5031 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081466    5031 flags.go:64] FLAG: --runonce="false"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081475    5031 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081479    5031 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081482    5031 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081485    5031 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081487    5031 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081490    5031 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081493    5031 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081497    5031 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081499    5031 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081502    5031 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081505    5031 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081508    5031 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081511    5031 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081514    5031 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081516    5031 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081522    5031 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081525    5031 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081527    5031 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081531    5031 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081573    5031 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:24:06.091200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081577    5031 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081580    5031 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081583    5031 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081586    5031 flags.go:64] FLAG: --v="2"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081595    5031 flags.go:64] FLAG: --version="false"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081601    5031 flags.go:64] FLAG: --vmodule=""
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081606    5031 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.081609    5031 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081781    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081786    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081790    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081793    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081795    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081799    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081801    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081804    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081807    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081809    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081812    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081815    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081817    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081820    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.091843 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081823    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081825    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081828    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081830    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081834    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081838    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081841    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081844    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081846    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081849    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081852    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081854    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081857    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081859    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081861    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081866    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081869    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081872    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081874    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.092386 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081877    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081879    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081882    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081884    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081887    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081890    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081893    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081896    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081898    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081901    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081904    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081906    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081909    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081911    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081913    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081916    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081918    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081921    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081923    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.092915 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081927    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081931    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081935    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081938    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081941    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081944    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081947    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081950    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081952    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081957    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081959    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081962    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081965    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081967    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081970    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081973    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081976    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081978    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081980    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081983    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.093438 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081986    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081989    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081992    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081994    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.081997    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082000    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082002    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082005    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082007    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082010    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082013    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082015    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082017    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082020    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082023    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082025    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082028    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082030    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082033    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082035    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.094033 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082038    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082042    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082047    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082050    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082052    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082055    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082057    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082060    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082062    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082064    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082067    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082070    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082079    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082081    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.082110    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082146    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.094682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082151    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082155    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082158    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082161    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082164    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082166    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082169    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082171    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082174    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082176    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082179    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082181    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082184    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082187    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082189    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082192    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082194    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082200    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082203    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082208    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.095454 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082211    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082214    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082217    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082220    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082222    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082225    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082228    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082230    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082233    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082241    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082244    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082247    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082250    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082252    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082254    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082257    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082259    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082262    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082265    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.096233 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082267    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082269    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082272    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082275    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082277    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082279    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082282    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082284    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082287    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082289    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082293    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082295    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082301    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082304    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082307    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082310    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082314    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082317    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082320    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.096943 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082322    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082325    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082328    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082330    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082340    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082342    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082345    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082347    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082350    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082353    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082355    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082358    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082360    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082363    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082366    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082368    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082371    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082373    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082375    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.097487 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082378    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082380    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082383    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082385    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082389    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082392    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082396    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082398    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082401    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082404    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082406    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082409    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082411    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082414    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082416    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082418    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082421    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082423    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082432    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082435    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.098025 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082438    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082440    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082443    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082445    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082448    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082451    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082453    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.082456    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.088350    5031 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.088372    5031 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.088393    5031 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.088398    5031 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088467    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088474    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088477    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088480    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088483    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088487    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088489    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.098516 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088492    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088495    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088498    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088501    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088504    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088506    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088509    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088512    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088514    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088516    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088520    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088523    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088525    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088528    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088531    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088533    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088558    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088562    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088566    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.098998 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088569    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088578    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088581    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088584    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088586    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088589    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088592    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088595    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088597    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088600    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088602    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088605    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088607    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088610    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088612    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088615    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088619    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088624    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088627    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.099471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088630    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088634    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088637    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088639    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088642    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088645    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088648    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088651    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088654    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088656    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088659    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088661    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088664    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088667    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088669    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088672    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088674    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088677    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088680    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088682    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.099968 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088685    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088687    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088690    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088692    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088695    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088697    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088700    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088702    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088705    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088707    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088710    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088712    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088716    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088721    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088724    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088727    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088730    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088733    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088735    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.100471 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088738    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088740    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088743    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088745    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088748    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088750    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088753    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088755    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088758    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088761    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088764    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088766    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088769    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088771    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088774    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088776    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088779    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088781    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088784    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088787    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.100953 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088789    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.088792    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.088826    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089013    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089018    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089021    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089024    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089027    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089030    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089032    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089035    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089038    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089040    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089043    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089046    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.101432 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089048    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089051    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089054    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089056    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089058    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089061    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089064    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089067    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089069    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089072    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089075    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089077    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089080    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089082    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089085    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089087    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089090    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089092    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089095    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089097    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.101861 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089100    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089103    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089105    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089108    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089110    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089113    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089116    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089121    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089124    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089126    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089129    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089132    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089134    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089136    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089139    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089141    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089144    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089147    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089149    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.102392 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089152    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089155    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089157    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089160    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089162    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089165    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089167    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089170    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089172    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089175    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089177    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089180    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089182    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089185    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089187    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089190    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089192    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089195    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089197    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089200    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.103080 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089205    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089207    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089210    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089213    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089215    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089218    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089220    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089222    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089225    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089227    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089232    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089236    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089239    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089242    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089245    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089248    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089251    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089254    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089257    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.104036 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089259    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089262    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089265    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089267    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089270    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089272    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089274    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089277    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089280    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089282    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089284    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089288    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089291    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089295    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089299    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089302    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.104677 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.089325    5031 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089360    5031 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089364    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089367    5031 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089369    5031 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089372    5031 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089375    5031 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089378    5031 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089380    5031 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089383    5031 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089386    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089388    5031 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089391    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089394    5031 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089397    5031 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089399    5031 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089402    5031 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089405    5031 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089407    5031 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089410    5031 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:24:06.105100 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089412    5031 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089415    5031 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089417    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089420    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089422    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089425    5031 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089427    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089430    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089432    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089435    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089438    5031 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089441    5031 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089444    5031 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089446    5031 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089449    5031 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089452    5031 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089454    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089457    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089460    5031 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089464    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:24:06.105633 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089467    5031 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089469    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089472    5031 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089474    5031 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089477    5031 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089479    5031 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089481    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089484    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089486    5031 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089488    5031 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089491    5031 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089493    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089496    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089498    5031 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089500    5031 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089503    5031 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089507    5031 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089511    5031 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089514    5031 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:24:06.106119 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089518    5031 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089521    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089524    5031 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089527    5031 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089530    5031 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089533    5031 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089555    5031 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089557    5031 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089560    5031 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089563    5031 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089566    5031 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089568    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089572    5031 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089575    5031 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089577    5031 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089580    5031 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089582    5031 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089586    5031 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089589    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089592    5031 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:24:06.106603 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089596    5031 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089598    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089601    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089603    5031 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089606    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089609    5031 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089611    5031 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089614    5031 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089616    5031 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089618    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089621    5031 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089623    5031 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089626    5031 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089628    5031 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089631    5031 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089634    5031 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089638    5031 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089641    5031 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089644    5031 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:24:06.107078 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089647    5031 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089650    5031 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089652    5031 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089655    5031 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089657    5031 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089661    5031 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089663    5031 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089666    5031 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:06.089668    5031 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.089739    5031 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.090769    5031 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.090831    5031 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.091022    5031 server.go:1014] "Starting client certificate rotation"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.091166    5031 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.091226    5031 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:55 +0000 UTC" deadline="2028-08-01 19:08:23.131648062 +0000 UTC"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.091254    5031 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15332h44m17.040395769s"
Nov 01 22:24:06.107549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.091468    5031 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:24:06.107963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.092720    5031 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:24:06.107963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.094026    5031 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:24:06.107963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.096167    5031 log.go:25] "Validated CRI v1 image API"
Nov 01 22:24:06.107963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.096958    5031 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:24:06.107963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.098913    5031 fs.go:135] Filesystem UUIDs: map[6cf40e16-d200-447e-80c5-16ce70ac3c77:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89b5243b-12d2-4764-bc56-c896d0f01c4f:/dev/nvme0n1p3]
Nov 01 22:24:06.108109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.098932    5031 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cf76dc49297405baa7aaa49c78e0db0bdaae87769822760b5ea1d9c4405ce79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cf76dc49297405baa7aaa49c78e0db0bdaae87769822760b5ea1d9c4405ce79/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a58c23e3e29ff833f6fe08c0959b0f56fb9b3dbf9d2cd5c533c9166d900d8a58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a58c23e3e29ff833f6fe08c0959b0f56fb9b3dbf9d2cd5c533c9166d900d8a58/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~projected/kube-api-access-zbrmt:{mountpoint:/var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~projected/kube-api-access-zbrmt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/kube-api-access-v9slz:{mountpoint:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/kube-api-access-v9slz major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/registry-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~projected/kube-api-access-f9mzv:{mountpoint:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~projected/kube-api-access-f9mzv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e3a1880-066c-4255-9da2-d6fad273bc48/volumes/kubernetes.io~projected/kube-api-access-cvxbq:{mountpoint:/var/lib/kubelet/pods/4e3a1880-066c-4255-9da2-d6fad273bc48/volumes/kubernetes.io~projected/kube-api-access-cvxbq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f0a8880-fd76-4eba-a768-a5e9577e986c/volumes/kubernetes.io~projected/kube-api-access-gbjcd:{mountpoint:/var/lib/kubelet/pods/6f0a8880-fd76-4eba-a768-a5e9577e986c/volumes/kubernetes.io~projected/kube-api-access-gbjcd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~projected/kube-api-access-br79m:{mountpoint:/var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~projected/kube-api-access-br79m major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/919cce7a-c6b0-4724-9b16-6314caf4716b/volumes/kubernetes.io~projected/kube-api-access-cccjg:{mountpoint:/var/lib/kubelet/pods/919cce7a-c6b0-4724-9b16-6314caf4716b/volumes/kubernetes.io~projected/kube-api-access-cccjg major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95d3c296-d0fc-4987-bc94-0ff8ab739120/volumes/kubernetes.io~projected/kube-api-access-sk465:{mountpoint:/var/lib/kubelet/pods/95d3c296-d0fc-4987-bc94-0ff8ab739120/volumes/kubernetes.io~projected/kube-api-access-sk465 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96be3757-1e1a-40a2-95f8-81b9193ef313/volumes/kubernetes.io~projected/kube-api-access-mmjsw:{mountpoint:/var/lib/kubelet/pods/96be3757-1e1a-40a2-95f8-81b9193ef313/volumes/kubernetes.io~projected/kube-api-access-mmjsw major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~projected/kube-api-access-zh77z:{mountpoint:/var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~projected/kube-api-access-zh77z major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~secret/metrics-certs major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21a67f5-fb49-4af9-90b2-b68da8775bd6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a21a67f5-fb49-4af9-90b2-b68da8775bd6/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~projected/kube-api-access-dlf8f:{mountpoint:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~projected/kube-api-access-dlf8f major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~projected/kube-api-access-rf5xp:{mountpoint:/var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~projected/kube-api-access-rf5xp major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~projected/kube-api-access-rrgwh:{mountpoint:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~projected/kube-api-access-rrgwh major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~projected/kube-api-access-qg2qh:{mountpoint:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~projected/kube-api-access-qg2qh major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2cf3107-3278-46ee-8ebd-165aa8bba436/volumes/kubernetes.io~projected/kube-api-access-jxt2k:{mountpoint:/var/lib/kubelet/pods/d2cf3107-3278-46ee-8ebd-165aa8bba436/volumes/kubernetes.io~projected/kube-api-access-jxt2k major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a/volumes/kubernetes.io~projected/kube-api-access-pg7wv:{mountpoint:/var/lib/kubelet/pods/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a/volumes/kubernetes.io~projected/kube-api-access-pg7wv major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f76d2c9f-944d-40e7-a5be-23725a4575b2/volumes/kubernetes.io~projected/kube-api-access-gf2qn:{mountpoint:/var/lib/kubelet/pods/f76d2c9f-944d-40e7-a5be-23725a4575b2/volumes/kubernetes.io~projected/kube-api-access-gf2qn major:0 minor:53 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/75fbf493d0336cb916c7d092981c257dd0e78a1b645f30d1ab6de1ee1b6894a6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/706b0173a1de6a1afedf0f92408444634ea149ab7b1aac2571ef6ab0f0f8c9d7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/16baffbd06b418b740894b17d15e7046f53f8b0adf3afd4445fbcece4198364f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7682744fb9648c2a275371064a8c43c07c891e0484639d8f20ef8b39e6e3a6be/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3070d31a7bbcb9a2e77dd286d21fa5403c49275f35554d2fc52dfc0d9771dbf5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/51a16a7d6e457bcf5e0f4d46838b0e6cf708f27a601f074505d2329a31a358f1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e4fe00731be4f429626c869e722cdc3a025d9b79470ea042573597098ea07074/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b0fbd33608c3a07d7b23661d3679866eabcdd515ce0df60aaee4603de9d48045/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/fa5c695260850e83decf9a52490b75a325e241e4a45c2d672eb240ca3a493fb4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/208fb79dc228a0a6f4f950e96a0428ef93d831840c3459bbdd32e7ed73d46ac5/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/ec61643a76cae624a87238a031444421d5b5b2df72755e5a6c2c9727fba916fb/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/b7ad692124b33667e36270885fb64facf97480111c97b2ae4879faa894cb0a67/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/93623cd3bbb37b056c149c461877692475cb30422914b52ae90e229628bc0bc2/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/17c4cc2a489dbf0ee0276bb53bfa55b160619be33ff1ad8774a347a8662388ae/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/0ce6663fa520f36bdb9570554b769a4964866ad0c94384b4d8f269f2746957d1/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2350eedecb5cf76a0896f5680964510bb6fc8ac880907a7e006f11a26adc45e5/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1939b860b6eaf6bf1f20a74ee077d98e2f3b52dc95f0615e447eebaf5a2b06e8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b74f09562576f2ceed63d55dadef5bd5994c624f6006d8f872d0cab8c64ae47f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c30a4f960f19c47a21833d2e79176f591c78c1b64599402392e3a33fb1b6390c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e4f66c677541dccb39c96e008446cb801829530956f3283bb8c4a690284eee3f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7e41a939464f661e7057e05cbea46ea12feda734c471a79537f765f5d323a0a4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/521f0b56890268f0f5f8aa4fc52ed707bcdcf083e7f11a4fb66320d188cbbbfa/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/149e907c368812d7bc03329e2f10be290af30b65d725d04b63292e689b33e2c5/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/fec896818f61cd7f83790ea32fe666558fa161f1341bb441f781647e26c061fb/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8dbb4bec528995d7e09a56fcb72678842bc9eab24292622aed67840589045f0c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/778ac96b6e56989686e16315fd87e1a8aea90f28e351d8111ad94f8e11f57c77/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/d3b6a9bc398b56357e90c733c385c23408f6b2776fc9f2d2fe68ceb1a7682a71/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/de8518c99764bc95c99f6a241f1ad9c6c2d3dc9a9dafebe265113d49804d3775/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/48c7de2d9a4e60e76be12e585fd9d40f7836f9bc27519677d73e8cca92dde982/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/94aafacad72b4bb4542b045999a452debd5ea6ba202261c2de654dd44abfaaa0/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0c1499b7a9d33aac803c69bb9cf3e075b2d7e51654287c41d43ac6287255c899/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f9f6f9e033fe40058d10b3c1b1d360f684f2ef96e038b280e17c1197db31eddb/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/fe95b815785ee081dd205ede76d2952ad44305796e8e04a9c8de843ae4d8587d/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/1b9ca246ff4a8d01a35f485ea9d76e5535f459ed9fe52559fb95e39766c6d8ab/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/c35904157eb0bb1131e4db517f59978935e5a18094d0c6336282b23c21a903ae/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/54b31d8bdf0c2b014e44c082b1412e0eb4ec1b54452151f17d317b79face3618/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/c2dbfa99208fa4c928c792ba3be12c34eb9de5a19d79bf27fb0da0fe96dcc15c/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/43cf400dea32acf0c6b5b070f56a40ee8065ce21fc28caa3534c3c42484d778b/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/dae33cec6a058f26600619f46030260a12145f8926228c8effecc268353d8e0e/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/65ae2cec76a590b883228485642bd0da8ad4b54b17d97da50d80fbfdb3963d75/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/bbbc5cb2e7e72fa1a6327e6f61f1fbf13be251b38976f9b5c72940e0c2ca92f2/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/8935fd5adcb0402ad72dfae8cde67eac2d4af6c6e5b48824fc97715defd5ec1d/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/445b0e5663ae1b606d744b5a849d69c465772d8b206f90a0c3191358ec89a421/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/533a5af2cd9a431d60003e9795f653b17df559e4f9b3245f12667d3ffea83270/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/b7c4e680934faaeef2dd7bf7713ea6fd2b24c44a25d0e6e807ff64d76a034609/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/00e04de9fca1d1f251a6730b9aceaed4ef63678e65748545fb29e15cc1e890f0/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/bad2c987d9ed0dd3de9d0e49e70658fd13d01d7d1cf9dc83af0d4eb1b06b5b8a/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b7f7506362e3bcac0134457974bec70c581bf464ff3b3e49241d348a52232187/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/49a6724fc789f13a8043a2ed7a0924cd3feef9d74189368831089ea2eddcc056/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/f7f9cbfd018347a8fcbe8721080325e7385aeafa2adf3d23a2ea3304ec1ae4f7/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/abfa7c3dd3f7b8db37bcd826210442294f131387ed0f2c7ddebfdfd4a2663e7f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ab7625b6fc55d00f43480fadf1c788512c07c805428080cf679e3148b0e7bf72/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/42f2a7001367f625ee0c88c2e1a4339b250e7866c4b7a2c263520615dd1f9b42/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5dade6c3b3277669494ba70795c2fbef85d5f699f14a737b63ed7fc681690984/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cc5a746bf48aef20a96ba914c8c571c43c14c36442ddf1a89daa1a74790d63bd/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 22:24:06.108432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108066    5031 manager.go:217] Machine: {Timestamp:2026-11-01 22:24:06.107572215 +0000 UTC m=+0.063612051 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099647 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c638dbd86fb64791b01c4aeb93c1c SystemUUID:ec2c638d-bd86-fb64-791b-01c4aeb93c1c BootID:548210c3-255a-4abe-a307-6a236f6516c6 Filesystems:[{Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f76d2c9f-944d-40e7-a5be-23725a4575b2/volumes/kubernetes.io~projected/kube-api-access-gf2qn DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/260546beac7fce9f853ee2013676caf9ed5e6a88c8dd342c15fe7ec9384a5d72/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~projected/kube-api-access-rf5xp DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:260 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/919cce7a-c6b0-4724-9b16-6314caf4716b/volumes/kubernetes.io~projected/kube-api-access-cccjg DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cf76dc49297405baa7aaa49c78e0db0bdaae87769822760b5ea1d9c4405ce79/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a/volumes/kubernetes.io~projected/kube-api-access-pg7wv DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcebb298e647cbe33e9acf06e3a7887fe098bf0d37ec71335582be1c4b813a7e/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a21a67f5-fb49-4af9-90b2-b68da8775bd6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~projected/kube-api-access-zh77z DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~projected/kube-api-access-qg2qh DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/96be3757-1e1a-40a2-95f8-81b9193ef313/volumes/kubernetes.io~projected/kube-api-access-mmjsw DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~projected/kube-api-access-rrgwh DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~projected/kube-api-access-zbrmt DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~projected/kube-api-access-dlf8f DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f0a8880-fd76-4eba-a768-a5e9577e986c/volumes/kubernetes.io~projected/kube-api-access-gbjcd DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab/volumes/kubernetes.io~projected/kube-api-access-f9mzv DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~projected/kube-api-access-br79m DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/427f15b3e8de2bd1025ff020aaebadc6aa7934eadcb91438e677c0be8924207c/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56a67b4e6373dd5e742543a6354ac6cb5281fcfe6352e02eccc0e4963ba959af/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56f9b24028385485c654abc1897bc8cc885751393d0b840f2465f03aef08b238/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e061e4e4da0ff3252109fd4a47797d8bfa257e74e980b69df41e72913cd81c9d/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62431cdcfd804e7b46f70731f53585bd80bc0122cbd67c1b149f0f0be0608103/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3f009553652628218b993d7f2a1016ae188426df54b5af565dbb7fdd07c3ca7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f070787ce4adfb078a388faf7466ce3fb2bd63f2246de4a51bfa819dcabab514/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16b30a168047c8a2cba4e4c4f46f7c0ef043eecfe4e4863e7e7417b3df0a1417/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32fb645adfb668b9e0e0749838156049cb8c74ca6309564bb53c22b0816a958d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e3a1880-066c-4255-9da2-d6fad273bc48/volumes/kubernetes.io~projected/kube-api-access-cvxbq DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a58c23e3e29ff833f6fe08c0959b0f56fb9b3dbf9d2cd5c533c9166d900d8a58/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cb0e0a3c3036441b4ff3937b11dbaca5c30a611dfe7ed16c0a8496a8325e707/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c842552-5f6f-4868-8b12-5c97e1e35eb8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95d3c296-d0fc-4987-bc94-0ff8ab739120/volumes/kubernetes.io~projected/kube-api-access-sk465 DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac262171-cc91-4552-ad95-7af735ac5450/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da29c9d11087e2f518b5d5a529579ffc1ad2d9174d283f1017202fa1ef03afcd/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b8a1f158-f781-40f4-a9c4-899b928dbf8a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33175d9207abc3011b1fb19b543eb642f8e074723a5c777f3972f4e3fb780941/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2cf3107-3278-46ee-8ebd-165aa8bba436/volumes/kubernetes.io~projected/kube-api-access-jxt2k DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c780c544-2eef-4d4c-8b46-b3f079ef2e72/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa7dbfa-8562-4396-9657-18862faf25dc/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9990533f9390296301e21e471fefd8f9ef9d683b044dd543146825f6162d6443/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22260f24-c7ba-4462-ab58-c49f4bbbf1b1/volumes/kubernetes.io~projected/kube-api-access-v9slz DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/907b53dd-5fe8-4df2-b5eb-d0e487157df3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:267 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f1440d407f431db2e3a8febf51d9d7a90ff96fda72edf83f87257ff47ad1561/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/766e9eaaf82ecffc7fb7d8a71c02401cef62d248ea200bb319230a9e1a14f04f/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1cf76dc49297405 MacAddress:0e:cb:31:b5:23:3b Speed:10000 Mtu:8901} {Name:260546beac7fce9 MacAddress:8a:97:2b:5c:5b:48 Speed:10000 Mtu:8901} {Name:33175d9207abc30 MacAddress:ca:f4:88:6f:58:9c Speed:10000 Mtu:8901} {Name:427f15b3e8de2bd MacAddress:ba:e1:e3:80:8c:2e Speed:10000 Mtu:8901} {Name:62431cdcfd804e7 MacAddress:e2:03:ee:44:fc:14 Speed:10000 Mtu:8901} {Name:8cb0e0a3c303644 MacAddress:d6:fc:d2:e9:4c:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fd:92:8f:5d Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:81:95:9f:26:aa Speed:0 Mtu:8901} {Name:da29c9d11087e2f MacAddress:5e:7c:5c:de:bb:87 Speed:10000 Mtu:8901} {Name:e061e4e4da0ff32 MacAddress:f6:a2:48:0a:d7:b1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fd:92:8f:5d Speed:0 Mtu:9001} {Name:fcebb298e647cbe MacAddress:42:80:c2:d5:34:c4 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:34:95:04:e9:80 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:34:88:29:d8:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:24:06.108779 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108437    5031 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:24:06.108779 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108489    5031 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:24:06.108841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108799    5031 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:24:06.108962 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108822    5031 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-155.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:24:06.109003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108964    5031 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:24:06.109003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108972    5031 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:24:06.109003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108982    5031 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:24:06.109003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.108998    5031 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:24:06.109114 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109100    5031 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:24:06.109273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109264    5031 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:24:06.109382 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109373    5031 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:24:06.109416 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109410    5031 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:24:06.109449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109420    5031 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:24:06.109449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109430    5031 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:24:06.109449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109440    5031 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:24:06.109449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.109449    5031 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:24:06.110106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.110089    5031 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:24:06.110106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.110101    5031 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:24:06.110661 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.110638    5031 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:24:06.119335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119314    5031 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:24:06.119335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119329    5031 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:24:06.119570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119487    5031 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 22:24:06.119795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119778    5031 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:24:06.119861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119819    5031 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:24:06.119962 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119950    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:24:06.120016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119970    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:24:06.120016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119980    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:24:06.120016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119990    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:24:06.120016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.119999    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:24:06.120016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120009    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120019    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120028    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120040    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120048    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120081    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120096    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120154    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:24:06.120249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.120164    5031 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:24:06.124456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.124435    5031 server.go:1263] "Started kubelet"
Nov 01 22:24:06.124608 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.124550    5031 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:24:06.125869 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.124962    5031 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:24:06.125869 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.125096    5031 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:24:06.125869 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.125448    5031 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:24:06.125139 ip-10-0-138-155 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:24:06.132551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.130836    5031 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:24:06.133884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.133862    5031 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.133888    5031 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.133950    5031 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-08-03 10:27:44.112313416 +0000 UTC"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.133973    5031 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15372h3m37.97834616s"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.134005    5031 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.134012    5031 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:24:06.134107 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.134012    5031 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:24:06.134732 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.134132    5031 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:24:06.134862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.134763    5031 factory.go:55] Registering systemd factory
Nov 01 22:24:06.134862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.134780    5031 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:24:06.135308 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135288    5031 factory.go:153] Registering CRI-O factory
Nov 01 22:24:06.135383 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135312    5031 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:24:06.135434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135391    5031 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:24:06.135434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135432    5031 factory.go:103] Registering Raw factory
Nov 01 22:24:06.135527 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135452    5031 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:24:06.135971 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.135957    5031 manager.go:319] Starting recovery of all containers
Nov 01 22:24:06.136577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136468    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab" volumeName="kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides" seLinuxMountContext=""
Nov 01 22:24:06.136577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136509    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa7dbfa-8562-4396-9657-18862faf25dc" volumeName="kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z" seLinuxMountContext=""
Nov 01 22:24:06.136577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136524    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21a67f5-fb49-4af9-90b2-b68da8775bd6" volumeName="kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136605    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136623    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95d3c296-d0fc-4987-bc94-0ff8ab739120" volumeName="kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136635    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96be3757-1e1a-40a2-95f8-81b9193ef313" volumeName="kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136647    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136658    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76d2c9f-944d-40e7-a5be-23725a4575b2" volumeName="kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136672    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907b53dd-5fe8-4df2-b5eb-d0e487157df3" volumeName="kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136685    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c780c544-2eef-4d4c-8b46-b3f079ef2e72" volumeName="kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136700    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136711    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab" volumeName="kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136736    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e3a1880-066c-4255-9da2-d6fad273bc48" volumeName="kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136755    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa7dbfa-8562-4396-9657-18862faf25dc" volumeName="kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136769    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c780c544-2eef-4d4c-8b46-b3f079ef2e72" volumeName="kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh" seLinuxMountContext=""
Nov 01 22:24:06.136776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136782    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136794    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136807    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d937c5d0-bdd4-4e16-8f01-51750bdc2c7a" volumeName="kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136820    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f76d2c9f-944d-40e7-a5be-23725a4575b2" volumeName="kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136832    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c780c544-2eef-4d4c-8b46-b3f079ef2e72" volumeName="kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136844    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2cf3107-3278-46ee-8ebd-165aa8bba436" volumeName="kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136856    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c842552-5f6f-4868-8b12-5c97e1e35eb8" volumeName="kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136867    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136878    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc" volumeName="kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136889    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136916    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e3a1880-066c-4255-9da2-d6fad273bc48" volumeName="kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136929    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a1f158-f781-40f4-a9c4-899b928dbf8a" volumeName="kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136942    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c780c544-2eef-4d4c-8b46-b3f079ef2e72" volumeName="kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136968    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0a8880-fd76-4eba-a768-a5e9577e986c" volumeName="kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136980    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab" volumeName="kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.136993    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0a8880-fd76-4eba-a768-a5e9577e986c" volumeName="kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137005    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0a8880-fd76-4eba-a768-a5e9577e986c" volumeName="kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137016    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907b53dd-5fe8-4df2-b5eb-d0e487157df3" volumeName="kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137029    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc" volumeName="kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137041    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="919cce7a-c6b0-4724-9b16-6314caf4716b" volumeName="kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137052    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a1f158-f781-40f4-a9c4-899b928dbf8a" volumeName="kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137063    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a1f158-f781-40f4-a9c4-899b928dbf8a" volumeName="kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137077    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c842552-5f6f-4868-8b12-5c97e1e35eb8" volumeName="kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137090    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab" volumeName="kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137105    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907b53dd-5fe8-4df2-b5eb-d0e487157df3" volumeName="kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137117    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb" volumeName="kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137130    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8a1f158-f781-40f4-a9c4-899b928dbf8a" volumeName="kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 22:24:06.137135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137142    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137159    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137171    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e3a1880-066c-4255-9da2-d6fad273bc48" volumeName="kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137185    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="907b53dd-5fe8-4df2-b5eb-d0e487157df3" volumeName="kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137209    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="919cce7a-c6b0-4724-9b16-6314caf4716b" volumeName="kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137221    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2cf3107-3278-46ee-8ebd-165aa8bba436" volumeName="kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137245    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137258    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22260f24-c7ba-4462-ab58-c49f4bbbf1b1" volumeName="kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137289    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21a67f5-fb49-4af9-90b2-b68da8775bd6" volumeName="kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137312    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137324    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137339    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc" volumeName="kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137367    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab" volumeName="kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137379    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0a8880-fd76-4eba-a768-a5e9577e986c" volumeName="kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137464    5031 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac262171-cc91-4552-ad95-7af735ac5450" volumeName="kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls" seLinuxMountContext=""
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137480    5031 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:24:06.138169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.137490    5031 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:24:06.161825 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.161778    5031 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:24:06.163196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.163169    5031 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:24:06.163335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.163205    5031 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:24:06.163335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.163227    5031 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:24:06.163335 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.163270    5031 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 22:24:06.175573 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.175506    5031 generic.go:356] "Generic (PLEG): container finished" podID="3d418d10f70ddaf0baab50de85c3c25b" containerID="7fb98ef48322557995f47f34f2626ea9da42fa2d3c7aa2964c817c9362441241" exitCode=0
Nov 01 22:24:06.194643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194603    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="44daf30590960209ea984969b2c655cb1bfad7898ea8b2b65d9a14f41af12397" exitCode=0
Nov 01 22:24:06.194643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194639    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="8bd962f49a1714e686f363eb15bd37903deea8ce61e93e76ed2c493120a1cf4a" exitCode=0
Nov 01 22:24:06.194643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194648    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="b53f1e9caacb3631ee1f56d3bd86627793fe20911c19be1bfbbba528591c8809" exitCode=0
Nov 01 22:24:06.194905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194658    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="a18d38ca0b6fbd68ca203ff484a3c83712810adb977c5f91c24e9b9e50999993" exitCode=0
Nov 01 22:24:06.194905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194668    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="64294f6adf671e2ff493db9b854fd2bd68b587a4a204ed6230897c22f7b82fd6" exitCode=0
Nov 01 22:24:06.194905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.194680    5031 generic.go:356] "Generic (PLEG): container finished" podID="6f0a8880-fd76-4eba-a768-a5e9577e986c" containerID="27513e9031d8c149ae877dc9b17ac9e34ab9e5d525a534dea5ead1f7ab49ff44" exitCode=0
Nov 01 22:24:06.220595 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.220571    5031 manager.go:324] Recovery completed
Nov 01 22:24:06.234764 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.234736    5031 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:24:06.238453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.238436    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.240711 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.240681    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.240821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.240718    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.240821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.240730    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.243982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.243962    5031 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:24:06.243982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.243977    5031 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:24:06.244120 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.243986    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.244120 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244001    5031 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244170    5031 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244187    5031 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244210    5031 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244216    5031 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244221    5031 policy_none.go:50] "Start"
Nov 01 22:24:06.244229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244230    5031 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:24:06.244409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.244241    5031 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:24:06.245413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245390    5031 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 22:24:06.245413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245411    5031 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 22:24:06.245570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245419    5031 policy_none.go:44] "Start"
Nov 01 22:24:06.245570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245456    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.245570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245491    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.245570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.245506    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.249808 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.249778    5031 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:24:06.252316 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252296    5031 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:24:06.252421 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.252322    5031 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:24:06.252421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252332    5031 server.go:87] "Starting device plugin registration server"
Nov 01 22:24:06.252692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252679    5031 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:24:06.252734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252695    5031 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:24:06.252776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252757    5031 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:24:06.252877 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252843    5031 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:24:06.252927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252884    5031 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:24:06.252927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252891    5031 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:24:06.253033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.252947    5031 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:24:06.252872071Z","UUID":"c24344bd-e611-4210-a52f-2931b169bb76","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:24:06.254372 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.254355    5031 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:24:06.254487 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.254380    5031 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:24:06.254558 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.254507    5031 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:24:06.254981 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.254870    5031 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-155.us-west-2.compute.internal\" not found"
Nov 01 22:24:06.263390 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.263351    5031 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"]
Nov 01 22:24:06.263559 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.263482    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.264404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.264382    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.264494 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.264423    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.264494 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.264440    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.264832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.264799    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.264972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.264942    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.266431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266412    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.266569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266445    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.266569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266459    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.266569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266414    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.266569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266524    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.266569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266558    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.266779 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.266719    5031 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.266826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266755    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerStarted","Data":"28bd7bed9f1cabb9cc11a5922c238af3847294ec2fd69908c93fd1cdaab88333"}
Nov 01 22:24:06.266826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266803    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerDied","Data":"7fb98ef48322557995f47f34f2626ea9da42fa2d3c7aa2964c817c9362441241"}
Nov 01 22:24:06.266826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266821    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"3d418d10f70ddaf0baab50de85c3c25b","Type":"ContainerStarted","Data":"843954f3a5da5208e51acee32a72f9804b32b715ce6e8ab7723921de5b84a9b2"}
Nov 01 22:24:06.266965 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266839    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"7f3192736cdd118e9e4731ee3fb56338","Type":"ContainerStarted","Data":"8a6a3264f3c9022925015e645dc990ca766182b6c09fd44de6934db96c5d9086"}
Nov 01 22:24:06.266965 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266857    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal" event={"ID":"7f3192736cdd118e9e4731ee3fb56338","Type":"ContainerStarted","Data":"a58c23e3e29ff833f6fe08c0959b0f56fb9b3dbf9d2cd5c533c9166d900d8a58"}
Nov 01 22:24:06.266965 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.266877    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.267684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.267669    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.267760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.267695    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.267760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.267706    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.267846 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.267828    5031 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-155.us-west-2.compute.internal\" not found" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.303731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.303689    5031 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:24:06.304627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.304606    5031 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:24:06.304818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.304684    5031 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 22:24:06.304818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.304711    5031 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:24:06.334804 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.334723    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.351270 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.351240    5031 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:24:06.353164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.353148    5031 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:24:06.354365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.354350    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:24:06.354430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.354383    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:24:06.354430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.354397    5031 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-155.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:24:06.354510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.354438    5031 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.395603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.395567    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:24:06.395741 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.395619    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.395741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.395634    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.405661 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.405624    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.405661 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.405667    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.405767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.405685    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.450085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.450056    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:24:06.450222 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:06.450109    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.455044 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.455019    5031 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.455106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.455095    5031 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506038    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506075    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506103    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7f3192736cdd118e9e4731ee3fb56338-config\") pod \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"7f3192736cdd118e9e4731ee3fb56338\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:06.506272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:06.506192    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d418d10f70ddaf0baab50de85c3c25b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" (UID: \"3d418d10f70ddaf0baab50de85c3c25b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:07.110408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.110367    5031 apiserver.go:52] "Watching apiserver"
Nov 01 22:24:07.218358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.218323    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:07.218562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.218489    5031 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:07.252206 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.252172    5031 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:24:07.252954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.252925    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-v5fjs","openshift-dns/dns-default-sddpb","openshift-multus/multus-additional-cni-plugins-2prlz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj","openshift-dns/node-resolver-x57zp","openshift-multus/multus-nsn92","openshift-cluster-node-tuning-operator/tuned-q7rnx","openshift-ingress-canary/ingress-canary-tv2zn","kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf","openshift-network-operator/iptables-alerter-w67hg","kube-system/konnectivity-agent-hxbq6","openshift-image-registry/image-registry-6ddd6b4776-stdwm","openshift-monitoring/prometheus-operator-9cd89b74c-7lctl","openshift-insights/insights-runtime-extractor-5bshs","kube-system/global-pull-secret-syncer-n7thn","openshift-multus/network-metrics-daemon-sdh5p","openshift-image-registry/node-ca-46rjc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal","openshift-monitoring/node-exporter-8d6wj","openshift-network-diagnostics/network-check-target-7flsn"]
Nov 01 22:24:07.274710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.274672    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:24:07.274861 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:07.274736    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:07.275158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.275142    5031 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:24:07.275225 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:07.275183    5031 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-155.us-west-2.compute.internal"
Nov 01 22:24:07.303876 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.303836    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.304028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.303917    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:24:07.304028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.303928    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:24:07.304028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.303936    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:24:07.304028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304019    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304081    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304109    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304131    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304135    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304154    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304164    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304225    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304231    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304237    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.304244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304227    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:24:07.304752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304231    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.304752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304283    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:24:07.304752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304298    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:24:07.304752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.304547    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.305333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.305318    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:24:07.305333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.305327    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.305454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.305330    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:24:07.305502 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.305484    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:24:07.305533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.305520    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.306077 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306060    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:24:07.306176 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306078    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.306404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306391    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.306751 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306738    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:24:07.306810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306794    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:24:07.306860 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306847    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:24:07.306913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306893    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:24:07.306913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306899    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:24:07.306971 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306941    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:24:07.306971 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306961    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:24:07.307037 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.306992    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:24:07.307139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.307127    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.308055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.308043    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.335814 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.335790    5031 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:24:07.362088 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362012    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 22:24:07.362340 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362324    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 22:24:07.362622 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362606    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:24:07.362698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362685    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:24:07.362887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362868    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 22:24:07.362943 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362909    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:24:07.363007 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362959    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 22:24:07.363007 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.362999    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jjz5d\""
Nov 01 22:24:07.363101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363042    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dqcqf\""
Nov 01 22:24:07.363101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363050    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 22:24:07.363101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363083    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:24:07.363225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363106    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:24:07.363225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363116    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:24:07.363225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363121    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bn2bj\""
Nov 01 22:24:07.363225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363184    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 22:24:07.363225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363209    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 22:24:07.363368 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363243    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:24:07.363368 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363301    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8bwdj\""
Nov 01 22:24:07.363427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.363374    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:24:07.402760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402731    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:24:07.402760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402732    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402732    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402760    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402820    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402865    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402881    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.402973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.402908    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.403196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403148    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.403196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403149    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 22:24:07.403287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403203    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.403287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403220    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:24:07.403287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403224    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:24:07.403287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403275    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:24:07.403654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403638    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:24:07.403724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403678    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 22:24:07.403917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.403903    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 22:24:07.411213 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411188    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.411320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411221    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.411320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411238    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2qh\" (UniqueName: \"kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.411320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411254    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.411320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411272    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.411454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411381    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-ca-trust-extracted\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.411454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411441    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.411509 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411462    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.411571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411526    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919cce7a-c6b0-4724-9b16-6314caf4716b-tmp-dir\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.411628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411565    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.411685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411643    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.411685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411673    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.411769 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411682    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-config\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.411769 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411691    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.411858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411767    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9slz\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.411858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.411800    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.412176 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412158    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a1f158-f781-40f4-a9c4-899b928dbf8a-data-volume\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.412287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412271    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-image-registry-private-configuration\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.412317 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412283    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.412317 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412304    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.412379 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412321    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.412379 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412349    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.412431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412379    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.412431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412404    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.412497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412435    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.412497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412456    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.412497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412486    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccjg\" (UniqueName: \"kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.412598 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.412628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412597    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.412663 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412647    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.412696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412667    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.412752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412700    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.412800 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412772    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.412847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412790    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.412847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412811    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.412942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412846    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.412942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412931    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.413038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412951    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-installation-pull-secrets\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.413038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.412965    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.413038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413010    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjsw\" (UniqueName: \"kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413076    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413091    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-tls\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413128    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413151    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c780c544-2eef-4d4c-8b46-b3f079ef2e72-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.413184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413169    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413210    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413251    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413284    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413342    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413376    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413379    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413399    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.413474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413426    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxt2k\" (UniqueName: \"kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413490    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2qn\" (UniqueName: \"kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413498    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fa7dbfa-8562-4396-9657-18862faf25dc-metrics-certs\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413620    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5xp\" (UniqueName: \"kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413669    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413717    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413745    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.413836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413776    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413868    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413898    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrmt\" (UniqueName: \"kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413925    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.413984    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/907b53dd-5fe8-4df2-b5eb-d0e487157df3-metrics-tls\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414015    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8a1f158-f781-40f4-a9c4-899b928dbf8a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414041    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414079    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414109    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414115    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f0a8880-fd76-4eba-a768-a5e9577e986c-cni-binary-copy\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.414172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414126    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414156    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414219    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414250    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414273    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c780c544-2eef-4d4c-8b46-b3f079ef2e72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414292    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414297    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f76d2c9f-944d-40e7-a5be-23725a4575b2-serviceca\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414316    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414353    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414372    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mzv\" (UniqueName: \"kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414388    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414404    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414422    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjcd\" (UniqueName: \"kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414444    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c842552-5f6f-4868-8b12-5c97e1e35eb8-cert\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:24:07.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414470    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh77z\" (UniqueName: \"kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:24:07.414978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414531    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.414978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414594    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.415332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.414626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.415430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415345    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-registry-certificates\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.415430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415351    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.415430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415391    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415414    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415662 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415439    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.415662 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-textfile\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415662 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415516    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415667    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.415792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415710    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.415792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415723    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415745    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415793    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415805    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-metrics-client-ca\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415826    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7wv\" (UniqueName: \"kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv\") pod \"migrator-66df5d496b-7qhgf\" (UID: \"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415865    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415900    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415898    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-original-pull-secret\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415933    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.415967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415964    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.416357 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.415987    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.416357 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416020    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.416357 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416059    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.416357 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416298    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-tuned\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.416525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416362    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-tls\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.416525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416370    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.416525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416409    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.416525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416449    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/907b53dd-5fe8-4df2-b5eb-d0e487157df3-tmp-dir\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416765    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-trusted-ca\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416797    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416847    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416846    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovn-node-metrics-cert\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416894    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416931    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxbq\" (UniqueName: \"kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416960    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf8f\" (UniqueName: \"kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.416990    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417035    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417064    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417081    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-cni-binary-copy\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417092    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgwh\" (UniqueName: \"kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417122    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417150    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417175    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417204    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417233    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417262    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417284    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417314    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417344    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417374    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417403    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417426    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417455    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417486    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417488    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-daemon-config\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417515    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417566    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br79m\" (UniqueName: \"kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417929    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-tmp\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.417992    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a21a67f5-fb49-4af9-90b2-b68da8775bd6-agent-certs\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418187    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-accelerators-collector-config\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418441    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2cf3107-3278-46ee-8ebd-165aa8bba436-iptables-alerter-script\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418484    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a21a67f5-fb49-4af9-90b2-b68da8775bd6-konnectivity-ca\") pod \"konnectivity-agent-hxbq6\" (UID: \"a21a67f5-fb49-4af9-90b2-b68da8775bd6\") " pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-ovnkube-script-lib\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418722    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/907b53dd-5fe8-4df2-b5eb-d0e487157df3-config-volume\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.419110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.418897    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-env-overrides\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.462726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.462689    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9slz\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-kube-api-access-v9slz\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.462917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.462854    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2qh\" (UniqueName: \"kubernetes.io/projected/c780c544-2eef-4d4c-8b46-b3f079ef2e72-kube-api-access-qg2qh\") pod \"prometheus-operator-9cd89b74c-7lctl\" (UID: \"c780c544-2eef-4d4c-8b46-b3f079ef2e72\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7lctl"
Nov 01 22:24:07.463099 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.463071    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccjg\" (UniqueName: \"kubernetes.io/projected/919cce7a-c6b0-4724-9b16-6314caf4716b-kube-api-access-cccjg\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.463603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.463581    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22260f24-c7ba-4462-ab58-c49f4bbbf1b1-bound-sa-token\") pod \"image-registry-6ddd6b4776-stdwm\" (UID: \"22260f24-c7ba-4462-ab58-c49f4bbbf1b1\") " pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:07.464655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.464565    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjsw\" (UniqueName: \"kubernetes.io/projected/96be3757-1e1a-40a2-95f8-81b9193ef313-kube-api-access-mmjsw\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.465044 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxt2k\" (UniqueName: \"kubernetes.io/projected/d2cf3107-3278-46ee-8ebd-165aa8bba436-kube-api-access-jxt2k\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.465121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465047    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2qn\" (UniqueName: \"kubernetes.io/projected/f76d2c9f-944d-40e7-a5be-23725a4575b2-kube-api-access-gf2qn\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.465121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrmt\" (UniqueName: \"kubernetes.io/projected/0c842552-5f6f-4868-8b12-5c97e1e35eb8-kube-api-access-zbrmt\") pod \"ingress-canary-tv2zn\" (UID: \"0c842552-5f6f-4868-8b12-5c97e1e35eb8\") " pod="openshift-ingress-canary/ingress-canary-tv2zn"
Nov 01 22:24:07.465121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465099    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5xp\" (UniqueName: \"kubernetes.io/projected/b8a1f158-f781-40f4-a9c4-899b928dbf8a-kube-api-access-rf5xp\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.465695 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjcd\" (UniqueName: \"kubernetes.io/projected/6f0a8880-fd76-4eba-a768-a5e9577e986c-kube-api-access-gbjcd\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.465948 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.465934    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mzv\" (UniqueName: \"kubernetes.io/projected/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-kube-api-access-f9mzv\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.466316 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.466299    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7wv\" (UniqueName: \"kubernetes.io/projected/d937c5d0-bdd4-4e16-8f01-51750bdc2c7a-kube-api-access-pg7wv\") pod \"migrator-66df5d496b-7qhgf\" (UID: \"d937c5d0-bdd4-4e16-8f01-51750bdc2c7a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7qhgf"
Nov 01 22:24:07.466658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.466644    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk465\" (UniqueName: \"kubernetes.io/projected/95d3c296-d0fc-4987-bc94-0ff8ab739120-kube-api-access-sk465\") pod \"network-check-target-7flsn\" (UID: \"95d3c296-d0fc-4987-bc94-0ff8ab739120\") " pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:24:07.466715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.466692    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh77z\" (UniqueName: \"kubernetes.io/projected/9fa7dbfa-8562-4396-9657-18862faf25dc-kube-api-access-zh77z\") pod \"network-metrics-daemon-sdh5p\" (UID: \"9fa7dbfa-8562-4396-9657-18862faf25dc\") " pod="openshift-multus/network-metrics-daemon-sdh5p"
Nov 01 22:24:07.467896 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.467868    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf8f\" (UniqueName: \"kubernetes.io/projected/ac262171-cc91-4552-ad95-7af735ac5450-kube-api-access-dlf8f\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.468000 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.467979    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxbq\" (UniqueName: \"kubernetes.io/projected/4e3a1880-066c-4255-9da2-d6fad273bc48-kube-api-access-cvxbq\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.468084 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.468067    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br79m\" (UniqueName: \"kubernetes.io/projected/907b53dd-5fe8-4df2-b5eb-d0e487157df3-kube-api-access-br79m\") pod \"dns-default-sddpb\" (UID: \"907b53dd-5fe8-4df2-b5eb-d0e487157df3\") " pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:07.468252 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.468236    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgwh\" (UniqueName: \"kubernetes.io/projected/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-kube-api-access-rrgwh\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.517940 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.517901    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.517940 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.517941    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.517966    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.517987    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518007    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518038    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-k8s-cni-cncf-io\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518038    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-netns\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518062    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-systemd-units\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518037    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-cnibin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518069    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2cf3107-3278-46ee-8ebd-165aa8bba436-host-slash\") pod \"iptables-alerter-w67hg\" (UID: \"d2cf3107-3278-46ee-8ebd-165aa8bba436\") " pod="openshift-network-operator/iptables-alerter-w67hg"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518109    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518118    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-system-cni-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518156    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518165    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/919cce7a-c6b0-4724-9b16-6314caf4716b-hosts-file\") pod \"node-resolver-x57zp\" (UID: \"919cce7a-c6b0-4724-9b16-6314caf4716b\") " pod="openshift-dns/node-resolver-x57zp"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518178    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518212    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-run\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518203    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518250    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518274    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-kubelet\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518292    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-system-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518302    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518277    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-etc-kubernetes\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518330    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518331    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-registration-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518345    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518366    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518385    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518401    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518404    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-var-lib-kubelet\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518414    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518431    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518435    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518452    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518473    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f76d2c9f-944d-40e7-a5be-23725a4575b2-host\") pod \"node-ca-46rjc\" (UID: \"f76d2c9f-944d-40e7-a5be-23725a4575b2\") " pod="openshift-image-registry/node-ca-46rjc"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518485    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518493    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-socket-dir-parent\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518487    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-lib-modules\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518525    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-etc-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518509    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518558    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518485    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-systemd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518561    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-kubelet\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518581    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-modprobe-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518600    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518640    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518652    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-netd\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518689    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518698    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518723    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518747    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.518739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518749    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-host\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518771    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518787    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-node-log\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518796    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518810    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-cni-bin\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518832    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-kubelet-config\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518835    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518840    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8a1f158-f781-40f4-a9c4-899b928dbf8a-crio-socket\") pod \"insights-runtime-extractor-5bshs\" (UID: \"b8a1f158-f781-40f4-a9c4-899b928dbf8a\") " pod="openshift-insights/insights-runtime-extractor-5bshs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518859    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-conf-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518871    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518886    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518900    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518910    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-device-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518917    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518931    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518943    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-sys-fs\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518945    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518965    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518978    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-run-netns\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518985    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-run-multus-certs\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.518981    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519016    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-bin\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519021    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519026    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-sys\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519041    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-ovn\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-host-slash\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519048    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-conf\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519045    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519075    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519107    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519143    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519153    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb-dbus\") pod \"global-pull-secret-syncer-n7thn\" (UID: \"af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb\") " pod="kube-system/global-pull-secret-syncer-n7thn"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519166    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519179    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysconfig\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519182    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-run-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519144    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-kubernetes\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519189    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519219    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-log-socket\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519221    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-root\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519219    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519251    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519251    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-sys\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519258    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-systemd\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519283    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519290    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac262171-cc91-4552-ad95-7af735ac5450-node-exporter-wtmp\") pod \"node-exporter-8d6wj\" (UID: \"ac262171-cc91-4552-ad95-7af735ac5450\") " pod="openshift-monitoring/node-exporter-8d6wj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519304    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519324    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-host-var-lib-cni-multus\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519352    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519378    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519389    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/96be3757-1e1a-40a2-95f8-81b9193ef313-socket-dir\") pod \"aws-ebs-csi-driver-node-fjkhj\" (UID: \"96be3757-1e1a-40a2-95f8-81b9193ef313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjkhj"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519377    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab-var-lib-openvswitch\") pod \"ovnkube-node-v5fjs\" (UID: \"2ee86ae0-eaac-4b60-bfc3-bbd995ef3cab\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-multus-cni-dir\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519410    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f0a8880-fd76-4eba-a768-a5e9577e986c-cnibin\") pod \"multus-additional-cni-plugins-2prlz\" (UID: \"6f0a8880-fd76-4eba-a768-a5e9577e986c\") " pod="openshift-multus/multus-additional-cni-plugins-2prlz"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519431    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519481    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3a1880-066c-4255-9da2-d6fad273bc48-hostroot\") pod \"multus-nsn92\" (UID: \"4e3a1880-066c-4255-9da2-d6fad273bc48\") " pod="openshift-multus/multus-nsn92"
Nov 01 22:24:07.519849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.519564    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc-etc-sysctl-d\") pod \"tuned-q7rnx\" (UID: \"ba42d5e6-f6fb-49ea-876b-8719a5c9c6bc\") " pod="openshift-cluster-node-tuning-operator/tuned-q7rnx"
Nov 01 22:24:07.564259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:07.564240    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:24:08.360936 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:08.360908    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:08.986153 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:08.986102    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:10.228739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.228706    5031 generic.go:356] "Generic (PLEG): container finished" podID="ac262171-cc91-4552-ad95-7af735ac5450" containerID="618e0bb567076effc734a780a1bd0e5dee2ae84e28339d892bf09957a6af6983" exitCode=0
Nov 01 22:24:10.228739 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.228746    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d6wj" event={"ID":"ac262171-cc91-4552-ad95-7af735ac5450","Type":"ContainerDied","Data":"618e0bb567076effc734a780a1bd0e5dee2ae84e28339d892bf09957a6af6983"}
Nov 01 22:24:10.252816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.252779    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"]
Nov 01 22:24:10.259435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.259412    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.310870 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.310844    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 22:24:10.311015 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.310994    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vpzrt\""
Nov 01 22:24:10.311806 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.311739    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"]
Nov 01 22:24:10.444321 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.444286    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbwj\" (UniqueName: \"kubernetes.io/projected/16c03e15-6fbc-4efd-905f-ef9321873204-kube-api-access-knbwj\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.444519 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.444332    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c03e15-6fbc-4efd-905f-ef9321873204-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.444634 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.444604    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"]
Nov 01 22:24:10.448450 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.448424    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.497795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.497701    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 22:24:10.497795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.497794    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 22:24:10.501122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.498417    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 22:24:10.501122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.498457    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-40ca5lvphhc7e\""
Nov 01 22:24:10.501122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.498507    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bpv7t\""
Nov 01 22:24:10.501122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.498627    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 22:24:10.502864 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.502846    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"]
Nov 01 22:24:10.545268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545229    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-client-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545273    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-metrics-server-audit-profiles\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545303    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-tls\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545320    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbww\" (UniqueName: \"kubernetes.io/projected/735f3a97-3941-4569-8ca4-cb8beb72112b-kube-api-access-8hbww\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545411    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbwj\" (UniqueName: \"kubernetes.io/projected/16c03e15-6fbc-4efd-905f-ef9321873204-kube-api-access-knbwj\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.545580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545474    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-client-certs\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545490    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/735f3a97-3941-4569-8ca4-cb8beb72112b-audit-log\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.545675 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545594    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c03e15-6fbc-4efd-905f-ef9321873204-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.545963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.545945    5031 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:24:10.548425 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.548403    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16c03e15-6fbc-4efd-905f-ef9321873204-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.598655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.598626    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbwj\" (UniqueName: \"kubernetes.io/projected/16c03e15-6fbc-4efd-905f-ef9321873204-kube-api-access-knbwj\") pod \"monitoring-plugin-77f4f76594-qcw24\" (UID: \"16c03e15-6fbc-4efd-905f-ef9321873204\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:10.624794 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:10.624754    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:24:10.646158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646127    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-client-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646170    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-metrics-server-audit-profiles\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646197    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-tls\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646212    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbww\" (UniqueName: \"kubernetes.io/projected/735f3a97-3941-4569-8ca4-cb8beb72112b-kube-api-access-8hbww\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646236    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-client-certs\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646267    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646288    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/735f3a97-3941-4569-8ca4-cb8beb72112b-audit-log\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.646887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.646859    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/735f3a97-3941-4569-8ca4-cb8beb72112b-audit-log\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.647136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.647107    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.647701 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.647678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/735f3a97-3941-4569-8ca4-cb8beb72112b-metrics-server-audit-profiles\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.649048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.649028    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-client-ca-bundle\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.649349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.649329    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-tls\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.649522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.649501    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/735f3a97-3941-4569-8ca4-cb8beb72112b-secret-metrics-server-client-certs\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.699285 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.699242    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbww\" (UniqueName: \"kubernetes.io/projected/735f3a97-3941-4569-8ca4-cb8beb72112b-kube-api-access-8hbww\") pod \"metrics-server-b56c5cc6f-zmpvr\" (UID: \"735f3a97-3941-4569-8ca4-cb8beb72112b\") " pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.760421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.760324    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:10.887576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:10.887513    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:11.177456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.177417    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"]
Nov 01 22:24:11.186020 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:11.185988    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735f3a97_3941_4569_8ca4_cb8beb72112b.slice/crio-0db8a51595d3092a5f162421cad655fac3253d3a23dccd9a1e2f4a1d2e4e40a2 WatchSource:0}: Error finding container 0db8a51595d3092a5f162421cad655fac3253d3a23dccd9a1e2f4a1d2e4e40a2: Status 404 returned error can't find the container with id 0db8a51595d3092a5f162421cad655fac3253d3a23dccd9a1e2f4a1d2e4e40a2
Nov 01 22:24:11.234180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.234141    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d6wj" event={"ID":"ac262171-cc91-4552-ad95-7af735ac5450","Type":"ContainerStarted","Data":"14ef33189fefbc353a20913448c5649c256c3e3145a7e0179fffdfd563e9a5eb"}
Nov 01 22:24:11.234650 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.234208    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8d6wj" event={"ID":"ac262171-cc91-4552-ad95-7af735ac5450","Type":"ContainerStarted","Data":"394e53cbebde6a3c19b41ba6ca5be2e23f34b6787f9ec7789aa4f2daa7d1e143"}
Nov 01 22:24:11.235352 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.235329    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr" event={"ID":"735f3a97-3941-4569-8ca4-cb8beb72112b","Type":"ContainerStarted","Data":"0db8a51595d3092a5f162421cad655fac3253d3a23dccd9a1e2f4a1d2e4e40a2"}
Nov 01 22:24:11.278769 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.278732    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"]
Nov 01 22:24:11.286458 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:11.286423    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c03e15_6fbc_4efd_905f_ef9321873204.slice/crio-b462763ecb2629d19875992cfc5905ca326ef5ffb99a2c9a8d167ac3de0f26a2 WatchSource:0}: Error finding container b462763ecb2629d19875992cfc5905ca326ef5ffb99a2c9a8d167ac3de0f26a2: Status 404 returned error can't find the container with id b462763ecb2629d19875992cfc5905ca326ef5ffb99a2c9a8d167ac3de0f26a2
Nov 01 22:24:11.988904 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:11.988876    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sddpb"
Nov 01 22:24:12.239958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:12.239878    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24" event={"ID":"16c03e15-6fbc-4efd-905f-ef9321873204","Type":"ContainerStarted","Data":"b462763ecb2629d19875992cfc5905ca326ef5ffb99a2c9a8d167ac3de0f26a2"}
Nov 01 22:24:12.290594 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:12.290565    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:12.290811 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:12.290783    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:24:12.962764 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:12.962729    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5fjs"
Nov 01 22:24:14.249388 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.249353    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr" event={"ID":"735f3a97-3941-4569-8ca4-cb8beb72112b","Type":"ContainerStarted","Data":"614d6bada3071c6d02ef84be5a60ad6d01149fcb6b41aa1469d48c611830357a"}
Nov 01 22:24:14.250997 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.250973    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24" event={"ID":"16c03e15-6fbc-4efd-905f-ef9321873204","Type":"ContainerStarted","Data":"a3c43b57c0e2c99d3ac0341420f49dd0142a7fcf1fa96d33a9790690eeb60c99"}
Nov 01 22:24:14.358884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.358823    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr" podStartSLOduration=2.173897658 podStartE2EDuration="4.358805235s" totalImagesPullingTime="2.184907577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:14.358403264 +0000 UTC m=+8.314443109" watchObservedRunningTime="2026-11-01 22:24:14.358805235 +0000 UTC m=+8.314845073"
Nov 01 22:24:14.464198 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.464130    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24" podStartSLOduration=2.437320318 podStartE2EDuration="4.464109277s" totalImagesPullingTime="2.026788959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:14.463349604 +0000 UTC m=+8.419389448" watchObservedRunningTime="2026-11-01 22:24:14.464109277 +0000 UTC m=+8.420149119"
Nov 01 22:24:14.538570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.538473    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:24:14.544328 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.544296    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.595216 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595182    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 22:24:14.595395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595226    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 22:24:14.595395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595237    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 22:24:14.595395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595254    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 22:24:14.595395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595283    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 22:24:14.595395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.595348    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dr8npenn1744a\""
Nov 01 22:24:14.641522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.641480    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 22:24:14.641721 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.641573    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8kfxf\""
Nov 01 22:24:14.641721 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.641575    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 22:24:14.679280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679239    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679295    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679368    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679425    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679447    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679461    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndd86\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679569    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679643    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679680    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679704    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679784    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679818    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.679889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679854    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679919    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679942    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.679994    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.680012    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.680026    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.680086    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.680132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.680105    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.688060 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688035    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 22:24:14.688192 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688046    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 22:24:14.688247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688119    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 22:24:14.688247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688205    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 22:24:14.688247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688131    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 22:24:14.688586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.688514    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 22:24:14.734324 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.734282    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 22:24:14.735153 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.735132    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:24:14.780818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780783    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.780818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780823    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780844    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780871    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780892    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780919    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780948    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.780970    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781384 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781358    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781507    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781581    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781621    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781650    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781674    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781680    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781719    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781741    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndd86\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781777    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781855    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781890    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781929    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.781996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.781954    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.782982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.782469    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.782982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.782641    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.784091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.783312    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.784091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.783896    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.784851 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.784824    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.784935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.784873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.785209 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.785181    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.785969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.785942    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.786441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.786412    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.786960 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.786926    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.787184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.787164    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.787306 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.787278    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.787430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.787409    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.787714 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.787692    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.787766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.787724    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.788269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.788252    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.836204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.836165    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndd86\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\") pod \"prometheus-k8s-0\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:14.874218 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:14.874174    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:15.253925 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:15.253828    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:24:15.350586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:15.350487    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:24:15.357185 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:24:15.357155    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61fae84_b6cf_4e61_8358_5a4f086c161e.slice/crio-6fa5174ddfaa162707a199baa93d56b7799691d0b8d9b6eb65ce333752fdb757 WatchSource:0}: Error finding container 6fa5174ddfaa162707a199baa93d56b7799691d0b8d9b6eb65ce333752fdb757: Status 404 returned error can't find the container with id 6fa5174ddfaa162707a199baa93d56b7799691d0b8d9b6eb65ce333752fdb757
Nov 01 22:24:16.258128 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:16.258084    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"6fa5174ddfaa162707a199baa93d56b7799691d0b8d9b6eb65ce333752fdb757"}
Nov 01 22:24:16.313474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:16.313443    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7flsn"
Nov 01 22:24:17.262601 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:17.262504    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3" exitCode=0
Nov 01 22:24:17.262601 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:17.262580    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3"}
Nov 01 22:24:17.453013 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:17.452979    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ddd6b4776-stdwm"
Nov 01 22:24:18.157057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:18.157014    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:18.157237 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:18.157170    5031 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:24:18.162299 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:18.162271    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qcw24"
Nov 01 22:24:21.279098 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:21.279007    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"}
Nov 01 22:24:21.279098 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:21.279075    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"}
Nov 01 22:24:24.162500 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.162417    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:24.295583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.295528    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"}
Nov 01 22:24:24.295774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.295643    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"}
Nov 01 22:24:24.295774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.295689    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"}
Nov 01 22:24:24.295774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.295723    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerStarted","Data":"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"}
Nov 01 22:24:24.416225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.416108    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.639179247 podStartE2EDuration="10.416092068s" totalImagesPullingTime="7.759855162s" totalInitContainerRuntime="17.057659ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:24.415209997 +0000 UTC m=+18.371249840" watchObservedRunningTime="2026-11-01 22:24:24.416092068 +0000 UTC m=+18.372131913"
Nov 01 22:24:24.823520 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.823486    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:24.875332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:24.875299    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:34.650374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:34.650341    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hxbq6"
Nov 01 22:24:40.625177 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:24:40.625145    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:24:44.829752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:44.829722    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:44.833548 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:44.833515    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b56c5cc6f-zmpvr"
Nov 01 22:24:53.546938 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:53.546888    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:53.566617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:53.566590    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:24:54.413951 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:24:54.413925    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:10.624443 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:10.624410    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:25:17.540516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.540480    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:17.541048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.540895    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="prometheus" containerID="cri-o://6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b" gracePeriod=600
Nov 01 22:25:17.541048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.540937    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-web" containerID="cri-o://c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e" gracePeriod=600
Nov 01 22:25:17.541048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.540991    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy" containerID="cri-o://032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90" gracePeriod=600
Nov 01 22:25:17.541048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.540931    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418" gracePeriod=600
Nov 01 22:25:17.541263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.541066    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="config-reloader" containerID="cri-o://4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0" gracePeriod=600
Nov 01 22:25:17.541263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:17.541108    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="thanos-sidecar" containerID="cri-o://b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233" gracePeriod=600
Nov 01 22:25:18.486772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486732    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418" exitCode=0
Nov 01 22:25:18.486772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486767    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90" exitCode=0
Nov 01 22:25:18.486772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486777    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233" exitCode=0
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486790    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0" exitCode=0
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486800    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b" exitCode=0
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486805    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"}
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486873    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"}
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486901    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"}
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486925    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"}
Nov 01 22:25:18.487063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.486947    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"}
Nov 01 22:25:18.796874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.796847    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:18.807798 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807770    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.807934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807863    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.807934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807891    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.807934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807911    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808090 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807938    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808090 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.807968    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808088    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.808196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808116    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808167    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808199    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808238    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808274    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808317    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808330    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808374    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808374    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808404    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808437    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808467    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808497    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808530    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.808618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808589    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.809190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.808987    5031 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.809190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.809007    5031 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.809190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.809022    5031 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-metrics-client-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.810058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.809972    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:25:18.811126 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.811094    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.811126 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.811100    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.811563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.811501    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "kube-api-access-ndd86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:25:18.811978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.811949    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.812353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.812323    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:18.813842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.813750    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:25:18.813842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.813785    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.813865    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.813880    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.813906    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814233 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.814214    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.814316    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:25:18.814394 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.814345    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.814984 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.814961    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.823339 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.823309    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:18.910075 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910036    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\") pod \"b61fae84-b6cf-4e61-8358-5a4f086c161e\" (UID: \"b61fae84-b6cf-4e61-8358-5a4f086c161e\") "
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910167    5031 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910179    5031 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-db\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910189    5031 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910198    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndd86\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-kube-api-access-ndd86\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910208    5031 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-grpc-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910218    5031 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910227    5031 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b61fae84-b6cf-4e61-8358-5a4f086c161e-tls-assets\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910235    5031 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-metrics-client-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910245    5031 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910254    5031 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-web-config\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910262    5031 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910270    5031 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-kube-rbac-proxy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910280    5031 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910288    5031 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b61fae84-b6cf-4e61-8358-5a4f086c161e-config-out\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910298    5031 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b61fae84-b6cf-4e61-8358-5a4f086c161e-config\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910306    5031 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:18.910464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:18.910388    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle" pod "b61fae84-b6cf-4e61-8358-5a4f086c161e" (UID: "b61fae84-b6cf-4e61-8358-5a4f086c161e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:19.010940 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.010906    5031 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b61fae84-b6cf-4e61-8358-5a4f086c161e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:19.494203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.494172    5031 generic.go:356] "Generic (PLEG): container finished" podID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerID="c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e" exitCode=0
Nov 01 22:25:19.494367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.494212    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"}
Nov 01 22:25:19.494367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.494260    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.494367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.494278    5031 scope.go:122] "RemoveContainer" containerID="82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"
Nov 01 22:25:19.494484 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.494269    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b61fae84-b6cf-4e61-8358-5a4f086c161e","Type":"ContainerDied","Data":"6fa5174ddfaa162707a199baa93d56b7799691d0b8d9b6eb65ce333752fdb757"}
Nov 01 22:25:19.503286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.503270    5031 scope.go:122] "RemoveContainer" containerID="032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"
Nov 01 22:25:19.511231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.511211    5031 scope.go:122] "RemoveContainer" containerID="c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"
Nov 01 22:25:19.518579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.518557    5031 scope.go:122] "RemoveContainer" containerID="b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"
Nov 01 22:25:19.527113 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.527094    5031 scope.go:122] "RemoveContainer" containerID="4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"
Nov 01 22:25:19.536710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.536689    5031 scope.go:122] "RemoveContainer" containerID="6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"
Nov 01 22:25:19.544407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.544388    5031 scope.go:122] "RemoveContainer" containerID="d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3"
Nov 01 22:25:19.551499 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.551478    5031 scope.go:122] "RemoveContainer" containerID="82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"
Nov 01 22:25:19.551847 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.551827    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418\": container with ID starting with 82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418 not found: ID does not exist" containerID="82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"
Nov 01 22:25:19.551899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.551858    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418"} err="failed to get container status \"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418\": rpc error: code = NotFound desc = could not find container \"82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418\": container with ID starting with 82cd68fbd3978791f46db34b3de91085b897fbc25976986011a3b7196e29c418 not found: ID does not exist"
Nov 01 22:25:19.551899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.551894    5031 scope.go:122] "RemoveContainer" containerID="032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"
Nov 01 22:25:19.552181 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.552159    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90\": container with ID starting with 032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90 not found: ID does not exist" containerID="032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"
Nov 01 22:25:19.552237 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552192    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90"} err="failed to get container status \"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90\": rpc error: code = NotFound desc = could not find container \"032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90\": container with ID starting with 032c1b1cc4f0772f5791c6959842c7225cd3baed453d2188fd4143451b6e7f90 not found: ID does not exist"
Nov 01 22:25:19.552237 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552209    5031 scope.go:122] "RemoveContainer" containerID="c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"
Nov 01 22:25:19.552454 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.552439    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e\": container with ID starting with c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e not found: ID does not exist" containerID="c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"
Nov 01 22:25:19.552504 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552459    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e"} err="failed to get container status \"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e\": rpc error: code = NotFound desc = could not find container \"c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e\": container with ID starting with c86036745dcbf1285a54cacdb5b4edb5eac8aa920e4d02c6388a5b0aef06014e not found: ID does not exist"
Nov 01 22:25:19.552504 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552474    5031 scope.go:122] "RemoveContainer" containerID="b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"
Nov 01 22:25:19.552839 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.552823    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233\": container with ID starting with b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233 not found: ID does not exist" containerID="b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"
Nov 01 22:25:19.552886 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552847    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233"} err="failed to get container status \"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233\": rpc error: code = NotFound desc = could not find container \"b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233\": container with ID starting with b7101a8b728fe95789581153b3fc346d17adcda1d706c29efac8267df1600233 not found: ID does not exist"
Nov 01 22:25:19.552886 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.552860    5031 scope.go:122] "RemoveContainer" containerID="4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"
Nov 01 22:25:19.553083 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.553068    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0\": container with ID starting with 4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0 not found: ID does not exist" containerID="4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"
Nov 01 22:25:19.553121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.553087    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0"} err="failed to get container status \"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0\": rpc error: code = NotFound desc = could not find container \"4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0\": container with ID starting with 4f7ff246e72ce580d856da314ef5e952dadfe12ca91bf0627999d7972910eca0 not found: ID does not exist"
Nov 01 22:25:19.553121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.553098    5031 scope.go:122] "RemoveContainer" containerID="6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"
Nov 01 22:25:19.553308 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.553291    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b\": container with ID starting with 6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b not found: ID does not exist" containerID="6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"
Nov 01 22:25:19.553346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.553313    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b"} err="failed to get container status \"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b\": rpc error: code = NotFound desc = could not find container \"6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b\": container with ID starting with 6d78463caacf48fb46f3d65b89cd499471c51e10f5cdfca4d830c2c5bb96459b not found: ID does not exist"
Nov 01 22:25:19.553346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.553326    5031 scope.go:122] "RemoveContainer" containerID="d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3"
Nov 01 22:25:19.553563 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:19.553527    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3\": container with ID starting with d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3 not found: ID does not exist" containerID="d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3"
Nov 01 22:25:19.553621 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.553562    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3"} err="failed to get container status \"d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3\": rpc error: code = NotFound desc = could not find container \"d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3\": container with ID starting with d45fcedfef8b9dc37ad1fc4905f1cae204165889653e4aad70e97d6ed84d69e3 not found: ID does not exist"
Nov 01 22:25:19.655291 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.655253    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:19.701497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.701469    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702030    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702404    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="init-config-reloader"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702420    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="init-config-reloader"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702436    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="prometheus"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702443    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="prometheus"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702458    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-thanos"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702465    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-thanos"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702480    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="config-reloader"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702486    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="config-reloader"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702496    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="thanos-sidecar"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702503    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="thanos-sidecar"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702515    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-web"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702521    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-web"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702530    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702556    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702618    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-thanos"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702630    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="prometheus"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702640    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="config-reloader"
Nov 01 22:25:19.702627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702648    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="thanos-sidecar"
Nov 01 22:25:19.703343 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702656    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy-web"
Nov 01 22:25:19.703343 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.702664    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" containerName="kube-rbac-proxy"
Nov 01 22:25:19.707906 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.707879    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716770 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716749    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716781    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716801    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716869    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716897    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.716958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716950    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717061 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.716985    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717092 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717063    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717092 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717080    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717194 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717094    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rnx\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-kube-api-access-n5rnx\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717194 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717111    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717213    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717261    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717276    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717292    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717315    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717346    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717492 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717416    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.717492 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.717453    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.757302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757214    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 22:25:19.757302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757240    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 22:25:19.757302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757256    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757380    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757432    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757460    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757497    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dr8npenn1744a\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757504    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757497    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757561    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757566    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 22:25:19.757629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.757629    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 22:25:19.803587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.803522    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 22:25:19.803587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.803526    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8kfxf\""
Nov 01 22:25:19.804259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.804236    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 22:25:19.804366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.804279    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 22:25:19.805081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.805063    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:19.818685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818648    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818686    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818708    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818728    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818745    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818770    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.818897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818798    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818967    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.818995    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819013    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819043    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819059    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819084    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819098    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819119    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rnx\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-kube-api-access-n5rnx\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819144    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819173    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819225    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819250    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819587    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.819859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819628    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.820108 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.819864    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.820843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.820807    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.822413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.821858    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.822413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.822340    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.823286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.823250    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.823405 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.823343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.823487 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.823402    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.823649 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.823628    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b432e902-88e9-414d-9203-b9215c810eda-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.824489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.824065    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-web-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.824489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.824108    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b432e902-88e9-414d-9203-b9215c810eda-config-out\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.824712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.824510    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.825618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.825086    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.825618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.825221    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.825618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.825287    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.825618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.825525    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-config\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.825884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.825696    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.826115 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.826090    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b432e902-88e9-414d-9203-b9215c810eda-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:19.874615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:19.874578    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rnx\" (UniqueName: \"kubernetes.io/projected/b432e902-88e9-414d-9203-b9215c810eda-kube-api-access-n5rnx\") pod \"prometheus-k8s-0\" (UID: \"b432e902-88e9-414d-9203-b9215c810eda\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:20.034439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:20.034350    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:20.167822 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:20.167784    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61fae84-b6cf-4e61-8358-5a4f086c161e" path="/var/lib/kubelet/pods/b61fae84-b6cf-4e61-8358-5a4f086c161e/volumes"
Nov 01 22:25:20.460809 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:20.460773    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 22:25:20.466616 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:25:20.466586    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb432e902_88e9_414d_9203_b9215c810eda.slice/crio-463e29a45c50f01e79b30307d16b485fdd744ba44d4edacab84b314ac9b0584a WatchSource:0}: Error finding container 463e29a45c50f01e79b30307d16b485fdd744ba44d4edacab84b314ac9b0584a: Status 404 returned error can't find the container with id 463e29a45c50f01e79b30307d16b485fdd744ba44d4edacab84b314ac9b0584a
Nov 01 22:25:20.500391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:20.500357    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"463e29a45c50f01e79b30307d16b485fdd744ba44d4edacab84b314ac9b0584a"}
Nov 01 22:25:21.505807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:21.505773    5031 generic.go:356] "Generic (PLEG): container finished" podID="b432e902-88e9-414d-9203-b9215c810eda" containerID="4ae7527f2124270c505debfaf58d5f92a29939ce106c8e0a7bbaa224462cfad5" exitCode=0
Nov 01 22:25:21.506220 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:21.505859    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerDied","Data":"4ae7527f2124270c505debfaf58d5f92a29939ce106c8e0a7bbaa224462cfad5"}
Nov 01 22:25:22.512368 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512330    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"e2935e40cee6870a343b8e6ae5499d61172852c233539f17e7ed7a80344f3e67"}
Nov 01 22:25:22.512760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512401    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"32e52ecbf0ea3ebbff118ebc495f78cccab5ac6c8e62abd4f6fe500b8ffc45e1"}
Nov 01 22:25:22.512760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512428    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"0744e54527a8f41030f0238149c69f3305af8749209277a30e2b000e7b9f3ae3"}
Nov 01 22:25:22.512760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"745796def091c9a6cc524ac40de3ff41fae6ba6624c97c5bd11c958994b0605d"}
Nov 01 22:25:22.512760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512475    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"ed7ee99c3a7ef3745fc8532e0eee7ff75ec424d82fcd500a469c3617ee6904d8"}
Nov 01 22:25:22.512760 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.512499    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b432e902-88e9-414d-9203-b9215c810eda","Type":"ContainerStarted","Data":"5c9ede5b0aa264666a7f41afbc10bb2f112ee7615b4665114ee4c23f307eca1b"}
Nov 01 22:25:22.628115 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:22.628049    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.61054325 podStartE2EDuration="3.628020412s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.477162ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:22.627467391 +0000 UTC m=+76.583507235" watchObservedRunningTime="2026-11-01 22:25:22.628020412 +0000 UTC m=+76.584060260"
Nov 01 22:25:25.034712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:25:25.034676    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:25:40.624724 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:25:40.624688    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:26:10.624228 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:26:10.624183    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:26:20.034723 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:26:20.034678    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:26:20.051184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:26:20.051159    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:26:20.727273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:26:20.727244    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 22:26:40.624841 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:26:40.624802    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:27:10.624560 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:27:10.624505    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:27:40.625305 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:27:40.625219    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:28:10.625126 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:28:10.625084    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:28:40.625132 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:28:40.625091    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:28:41.354766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.354729    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"]
Nov 01 22:28:41.358261 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.358221    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.407574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.407445    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 22:28:41.407822 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.407688    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 22:28:41.407822 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.407723    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 22:28:41.407953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.407847    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 22:28:41.407953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.407925    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 22:28:41.408033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.408015    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jfzq8\""
Nov 01 22:28:41.455032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.454991    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"]
Nov 01 22:28:41.495978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.495940    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.496140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.495980    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.496140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.496069    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.496140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.496112    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.496292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.496167    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.496292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.496223    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtplk\" (UniqueName: \"kubernetes.io/projected/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-kube-api-access-rtplk\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.596959    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597002    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597038    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597068    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597098    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597122    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtplk\" (UniqueName: \"kubernetes.io/projected/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-kube-api-access-rtplk\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.597553 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.597497    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.599740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.599712    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.599861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.599720    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.599901 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.599862    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.600005 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.599981    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.652437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.652355    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtplk\" (UniqueName: \"kubernetes.io/projected/b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b-kube-api-access-rtplk\") pod \"opendatahub-operator-controller-manager-57f65ddd4-gbh8j\" (UID: \"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:41.674102 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:41.674065    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:42.212882 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:42.212847    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"]
Nov 01 22:28:42.224994 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:28:42.224919    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eeadab_3ea1_4f09_b8a4_0e216e9b1e7b.slice/crio-2b1014255915d3b16fd5da9e07b57fb8046ec429b558902cf97b74699eaceabd WatchSource:0}: Error finding container 2b1014255915d3b16fd5da9e07b57fb8046ec429b558902cf97b74699eaceabd: Status 404 returned error can't find the container with id 2b1014255915d3b16fd5da9e07b57fb8046ec429b558902cf97b74699eaceabd
Nov 01 22:28:43.224813 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:43.224774    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j" event={"ID":"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b","Type":"ContainerStarted","Data":"2b1014255915d3b16fd5da9e07b57fb8046ec429b558902cf97b74699eaceabd"}
Nov 01 22:28:46.236609 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:46.236575    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j" event={"ID":"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b","Type":"ContainerStarted","Data":"3dabe198019acf798d15d33fcc5f6bb21f781f0f9bb20119c5bec3d52450f1cd"}
Nov 01 22:28:47.242846 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:47.242805    5031 generic.go:356] "Generic (PLEG): container finished" podID="b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b" containerID="3dabe198019acf798d15d33fcc5f6bb21f781f0f9bb20119c5bec3d52450f1cd" exitCode=0
Nov 01 22:28:47.243338 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:47.242898    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j" event={"ID":"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b","Type":"ContainerDied","Data":"3dabe198019acf798d15d33fcc5f6bb21f781f0f9bb20119c5bec3d52450f1cd"}
Nov 01 22:28:48.248248 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:48.248221    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j" event={"ID":"b8eeadab-3ea1-4f09-b8a4-0e216e9b1e7b","Type":"ContainerStarted","Data":"a87235d7b39d263b8704d1d60b1501cbf31a4d6e36a6a0697c5259dea3289f2a"}
Nov 01 22:28:48.248658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:48.248360    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:48.467050 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:48.466946    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j" podStartSLOduration=2.245082163 podStartE2EDuration="7.466931631s" totalImagesPullingTime="4.733607619s" totalInitContainerRuntime="488.241849ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:48.420648037 +0000 UTC m=+282.376687880" watchObservedRunningTime="2026-11-01 22:28:48.466931631 +0000 UTC m=+282.422971474"
Nov 01 22:28:57.828691 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.828653    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"]
Nov 01 22:28:57.836919 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.836893    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:57.884233 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.884192    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"]
Nov 01 22:28:57.929435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.929393    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 22:28:57.929633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.929442    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 22:28:57.929633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.929504    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j24mn\""
Nov 01 22:28:57.934236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.934209    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:57.934297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:57.934252    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsk6j\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-kube-api-access-jsk6j\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.035239 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.035198    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.035434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.035299    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsk6j\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-kube-api-access-jsk6j\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.094750 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.094678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.095403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.095380    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsk6j\" (UniqueName: \"kubernetes.io/projected/445c06f1-ac73-4779-9118-499a40730973-kube-api-access-jsk6j\") pod \"cert-manager-cainjector-5c57bbd4fc-7sxs6\" (UID: \"445c06f1-ac73-4779-9118-499a40730973\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.159590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.159520    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"
Nov 01 22:28:58.525794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:58.525716    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6"]
Nov 01 22:28:58.532746 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:28:58.532713    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod445c06f1_ac73_4779_9118_499a40730973.slice/crio-2fc635a0ed62819a612df12c7f074ccc16800b861a55a666c9d2ffba2aae6bb0 WatchSource:0}: Error finding container 2fc635a0ed62819a612df12c7f074ccc16800b861a55a666c9d2ffba2aae6bb0: Status 404 returned error can't find the container with id 2fc635a0ed62819a612df12c7f074ccc16800b861a55a666c9d2ffba2aae6bb0
Nov 01 22:28:59.254044 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:59.254014    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-gbh8j"
Nov 01 22:28:59.287812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:28:59.287776    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6" event={"ID":"445c06f1-ac73-4779-9118-499a40730973","Type":"ContainerStarted","Data":"2fc635a0ed62819a612df12c7f074ccc16800b861a55a666c9d2ffba2aae6bb0"}
Nov 01 22:29:03.304036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.303999    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6" event={"ID":"445c06f1-ac73-4779-9118-499a40730973","Type":"ContainerStarted","Data":"ed7a3ca38cac8e4eb785a7724ff00d79895f169dac1a2a6ac82752cc7fc9e511"}
Nov 01 22:29:03.331833 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.331793    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"]
Nov 01 22:29:03.334196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.334179    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.379454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.379418    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.379678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.379489    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhg9\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-kube-api-access-gbhg9\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.389308 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.389289    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pgmgg\""
Nov 01 22:29:03.407744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.407711    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"]
Nov 01 22:29:03.430417 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.430371    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7sxs6" podStartSLOduration=1.952281997 podStartE2EDuration="6.430357461s" totalImagesPullingTime="4.478075464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:03.42967355 +0000 UTC m=+297.385713393" watchObservedRunningTime="2026-11-01 22:29:03.430357461 +0000 UTC m=+297.386397305"
Nov 01 22:29:03.480084 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.480044    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhg9\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-kube-api-access-gbhg9\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.480259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.480102    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.533119 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.533091    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.533479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.533458    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhg9\" (UniqueName: \"kubernetes.io/projected/06afa0f3-960c-4468-8ad3-d0e6be05bc93-kube-api-access-gbhg9\") pod \"cert-manager-webhook-5dbbbf75cc-hxwjj\" (UID: \"06afa0f3-960c-4468-8ad3-d0e6be05bc93\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:03.644843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:03.644757    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:04.012769 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:04.012680    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"]
Nov 01 22:29:04.019591 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:04.019557    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06afa0f3_960c_4468_8ad3_d0e6be05bc93.slice/crio-c97e02644ddaa290bc744e1c0cefef4e8e96b6d2fbdc6683a2f3e64a9b1cba75 WatchSource:0}: Error finding container c97e02644ddaa290bc744e1c0cefef4e8e96b6d2fbdc6683a2f3e64a9b1cba75: Status 404 returned error can't find the container with id c97e02644ddaa290bc744e1c0cefef4e8e96b6d2fbdc6683a2f3e64a9b1cba75
Nov 01 22:29:04.309210 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:04.309172    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" event={"ID":"06afa0f3-960c-4468-8ad3-d0e6be05bc93","Type":"ContainerStarted","Data":"d62affbd04c91d9342a483a9d37c5c4988736fba9d1fbbf7c342847125f4baba"}
Nov 01 22:29:04.309210 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:04.309212    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" event={"ID":"06afa0f3-960c-4468-8ad3-d0e6be05bc93","Type":"ContainerStarted","Data":"c97e02644ddaa290bc744e1c0cefef4e8e96b6d2fbdc6683a2f3e64a9b1cba75"}
Nov 01 22:29:04.309756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:04.309318    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:04.415684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:04.415614    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" podStartSLOduration=1.4155954099999999 podStartE2EDuration="1.41559541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:04.41463859 +0000 UTC m=+298.370678434" watchObservedRunningTime="2026-11-01 22:29:04.41559541 +0000 UTC m=+298.371635254"
Nov 01 22:29:05.475051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.475014    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"]
Nov 01 22:29:05.477873 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.477852    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.497444 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.497414    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-tmp\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.497607 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.497461    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgm4n\" (UniqueName: \"kubernetes.io/projected/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-kube-api-access-cgm4n\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.526245 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.526214    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:29:05.572364 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.572326    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 22:29:05.572595 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.572575    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-pwwn2\""
Nov 01 22:29:05.573421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.573401    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"]
Nov 01 22:29:05.598734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.598696    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-tmp\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.598917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.598752    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgm4n\" (UniqueName: \"kubernetes.io/projected/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-kube-api-access-cgm4n\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.599179 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.599161    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-tmp\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.653438 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.653400    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgm4n\" (UniqueName: \"kubernetes.io/projected/c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717-kube-api-access-cgm4n\") pod \"jobset-operator-674cd59dc5-nqb4j\" (UID: \"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:05.789266 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:05.789186    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"
Nov 01 22:29:06.156984 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:06.156953    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j"]
Nov 01 22:29:06.166114 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:06.166077    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4beeeb0_c1ea_4498_b76e_f0ba3a3c8717.slice/crio-a94bd1932c940db5779b9b5ab3f46baf711d7c5a73e7e9404df0526e1d5e80a4 WatchSource:0}: Error finding container a94bd1932c940db5779b9b5ab3f46baf711d7c5a73e7e9404df0526e1d5e80a4: Status 404 returned error can't find the container with id a94bd1932c940db5779b9b5ab3f46baf711d7c5a73e7e9404df0526e1d5e80a4
Nov 01 22:29:06.317228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:06.317199    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j" event={"ID":"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717","Type":"ContainerStarted","Data":"a94bd1932c940db5779b9b5ab3f46baf711d7c5a73e7e9404df0526e1d5e80a4"}
Nov 01 22:29:10.315712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:10.315678    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj"
Nov 01 22:29:10.628511 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:29:10.628477    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:29:11.339820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:11.339783    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j" event={"ID":"c4beeeb0-c1ea-4498-b76e-f0ba3a3c8717","Type":"ContainerStarted","Data":"8c6b970016c8856bd35043bf7cca147e8bc068357c8facdda3497145439837fa"}
Nov 01 22:29:11.450375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:11.450317    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nqb4j" podStartSLOduration=2.088148455 podStartE2EDuration="6.450301685s" totalImagesPullingTime="4.36215323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:11.449020196 +0000 UTC m=+305.405060041" watchObservedRunningTime="2026-11-01 22:29:11.450301685 +0000 UTC m=+305.406341529"
Nov 01 22:29:18.589337 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.589300    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"]
Nov 01 22:29:18.592476 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.592457    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.597833 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.597804    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/501fa75d-cc43-478c-8d1c-d85b342bf25e-tmp\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.597979 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.597924    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501fa75d-cc43-478c-8d1c-d85b342bf25e-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.597979 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.597952    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbnb\" (UniqueName: \"kubernetes.io/projected/501fa75d-cc43-478c-8d1c-d85b342bf25e-kube-api-access-2fbnb\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.686984 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.686929    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 22:29:18.686984 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.686953    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 22:29:18.687268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.687031    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 22:29:18.687268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.687040    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7w5mv\""
Nov 01 22:29:18.687268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.687210    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 22:29:18.688249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.688221    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"]
Nov 01 22:29:18.698555 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.698510    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501fa75d-cc43-478c-8d1c-d85b342bf25e-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.698726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.698573    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbnb\" (UniqueName: \"kubernetes.io/projected/501fa75d-cc43-478c-8d1c-d85b342bf25e-kube-api-access-2fbnb\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.698726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.698617    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/501fa75d-cc43-478c-8d1c-d85b342bf25e-tmp\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.701147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.701119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/501fa75d-cc43-478c-8d1c-d85b342bf25e-tmp\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.701361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.701343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501fa75d-cc43-478c-8d1c-d85b342bf25e-tls-certs\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.751602 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.751570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbnb\" (UniqueName: \"kubernetes.io/projected/501fa75d-cc43-478c-8d1c-d85b342bf25e-kube-api-access-2fbnb\") pod \"kube-auth-proxy-59b78b6bf6-xwnhc\" (UID: \"501fa75d-cc43-478c-8d1c-d85b342bf25e\") " pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:18.904617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:18.904499    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"
Nov 01 22:29:19.271742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:19.271672    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc"]
Nov 01 22:29:19.278207 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:19.278177    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501fa75d_cc43_478c_8d1c_d85b342bf25e.slice/crio-8c4d62b1fa510bfa710bd60ddd0309e8afca435f7bbd619dbac1615f55dee6d5 WatchSource:0}: Error finding container 8c4d62b1fa510bfa710bd60ddd0309e8afca435f7bbd619dbac1615f55dee6d5: Status 404 returned error can't find the container with id 8c4d62b1fa510bfa710bd60ddd0309e8afca435f7bbd619dbac1615f55dee6d5
Nov 01 22:29:19.280447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:19.280425    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:29:19.371854 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:19.371820    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc" event={"ID":"501fa75d-cc43-478c-8d1c-d85b342bf25e","Type":"ContainerStarted","Data":"8c4d62b1fa510bfa710bd60ddd0309e8afca435f7bbd619dbac1615f55dee6d5"}
Nov 01 22:29:23.389664 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.389629    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc" event={"ID":"501fa75d-cc43-478c-8d1c-d85b342bf25e","Type":"ContainerStarted","Data":"a2ad7c4b6756b17d5c5a994104dbddd81b626573fdecad2570d3f556eede4daa"}
Nov 01 22:29:23.420736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.420697    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:23.423979 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.423956    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.433037 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.433013    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ldj\" (UniqueName: \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.433155 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.433065    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.486395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.486360    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 22:29:23.486395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.486360    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qf6wd\""
Nov 01 22:29:23.486620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.486401    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 22:29:23.486620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.486405    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 22:29:23.487319 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.487300    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:23.525373 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.525322    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59b78b6bf6-xwnhc" podStartSLOduration=2.287487665 podStartE2EDuration="5.525308047s" totalImagesPullingTime="3.237820382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:23.5243226 +0000 UTC m=+317.480362445" watchObservedRunningTime="2026-11-01 22:29:23.525308047 +0000 UTC m=+317.481347890"
Nov 01 22:29:23.533582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.533526    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ldj\" (UniqueName: \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.533770 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.533608    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.534224 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.534204    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.587244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.587211    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ldj\" (UniqueName: \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\") pod \"limitador-limitador-74dfcfb894-kv8hd\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:23.654580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.654484    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:23.735476 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:23.735429    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:24.101890 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:24.101859    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:24.108697 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:24.108669    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109bccba_5b18_412b_b6fd_09f4e49b64f6.slice/crio-73b2b2eb9933ac57f1682d140f04f27a13c40637a346d1e1c2f1a95ba5b91598 WatchSource:0}: Error finding container 73b2b2eb9933ac57f1682d140f04f27a13c40637a346d1e1c2f1a95ba5b91598: Status 404 returned error can't find the container with id 73b2b2eb9933ac57f1682d140f04f27a13c40637a346d1e1c2f1a95ba5b91598
Nov 01 22:29:24.394720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:24.394634    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" event={"ID":"109bccba-5b18-412b-b6fd-09f4e49b64f6","Type":"ContainerStarted","Data":"73b2b2eb9933ac57f1682d140f04f27a13c40637a346d1e1c2f1a95ba5b91598"}
Nov 01 22:29:27.650795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.650759    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"]
Nov 01 22:29:27.661034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.661006    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.668311 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.668281    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f599a277-3519-4cbc-b948-7d6ada3183dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.668458 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.668325    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f599a277-3519-4cbc-b948-7d6ada3183dc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.668458 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.668345    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq9x\" (UniqueName: \"kubernetes.io/projected/f599a277-3519-4cbc-b948-7d6ada3183dc-kube-api-access-9rq9x\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.706637 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.706554    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"]
Nov 01 22:29:27.751775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.751726    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 22:29:27.751940 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.751783    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 22:29:27.768873 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.768839    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f599a277-3519-4cbc-b948-7d6ada3183dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.769057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.768884    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f599a277-3519-4cbc-b948-7d6ada3183dc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.769057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.768912    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq9x\" (UniqueName: \"kubernetes.io/projected/f599a277-3519-4cbc-b948-7d6ada3183dc-kube-api-access-9rq9x\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.770157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.770125    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f599a277-3519-4cbc-b948-7d6ada3183dc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.772320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.772290    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f599a277-3519-4cbc-b948-7d6ada3183dc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.823393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.823361    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq9x\" (UniqueName: \"kubernetes.io/projected/f599a277-3519-4cbc-b948-7d6ada3183dc-kube-api-access-9rq9x\") pod \"kuadrant-console-plugin-66c4fb575d-krsjt\" (UID: \"f599a277-3519-4cbc-b948-7d6ada3183dc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:27.974223 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:27.974138    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"
Nov 01 22:29:28.341481 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:28.341449    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt"]
Nov 01 22:29:28.348150 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:28.348115    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf599a277_3519_4cbc_b948_7d6ada3183dc.slice/crio-5c7be18822a88b0fd576acec4e951e49b63d7bf7d254dac556a99f351053fcad WatchSource:0}: Error finding container 5c7be18822a88b0fd576acec4e951e49b63d7bf7d254dac556a99f351053fcad: Status 404 returned error can't find the container with id 5c7be18822a88b0fd576acec4e951e49b63d7bf7d254dac556a99f351053fcad
Nov 01 22:29:28.413457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:28.413415    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt" event={"ID":"f599a277-3519-4cbc-b948-7d6ada3183dc","Type":"ContainerStarted","Data":"5c7be18822a88b0fd576acec4e951e49b63d7bf7d254dac556a99f351053fcad"}
Nov 01 22:29:28.414879 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:28.414856    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" event={"ID":"109bccba-5b18-412b-b6fd-09f4e49b64f6","Type":"ContainerStarted","Data":"511c8994d623cf1192ce3bb6da4438448944bf91b55aedf5d2a1e4471bb68823"}
Nov 01 22:29:28.415000 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:28.414988    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:28.523363 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:28.523310    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" podStartSLOduration=1.56463414 podStartE2EDuration="5.523289019s" totalImagesPullingTime="3.958654879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:28.522455659 +0000 UTC m=+322.478495503" watchObservedRunningTime="2026-11-01 22:29:28.523289019 +0000 UTC m=+322.479328888"
Nov 01 22:29:36.450045 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:36.450010    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt" event={"ID":"f599a277-3519-4cbc-b948-7d6ada3183dc","Type":"ContainerStarted","Data":"267646cd5125d3b98a9a855c5d0a28f1801ea75d2e089573c636d23a3b71be91"}
Nov 01 22:29:36.556666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:36.556614    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-krsjt" podStartSLOduration=2.15584027 podStartE2EDuration="9.55659971s" totalImagesPullingTime="7.40075944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:36.555455512 +0000 UTC m=+330.511495356" watchObservedRunningTime="2026-11-01 22:29:36.55659971 +0000 UTC m=+330.512639553"
Nov 01 22:29:39.419958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:39.419927    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:40.625747 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:29:40.625704    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:29:44.055074 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.055034    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:44.055477 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.055246    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" podUID="109bccba-5b18-412b-b6fd-09f4e49b64f6" containerName="limitador" containerID="cri-o://511c8994d623cf1192ce3bb6da4438448944bf91b55aedf5d2a1e4471bb68823" gracePeriod=30
Nov 01 22:29:44.487604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.487532    5031 generic.go:356] "Generic (PLEG): container finished" podID="109bccba-5b18-412b-b6fd-09f4e49b64f6" containerID="511c8994d623cf1192ce3bb6da4438448944bf91b55aedf5d2a1e4471bb68823" exitCode=0
Nov 01 22:29:44.487604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.487580    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" event={"ID":"109bccba-5b18-412b-b6fd-09f4e49b64f6","Type":"ContainerDied","Data":"511c8994d623cf1192ce3bb6da4438448944bf91b55aedf5d2a1e4471bb68823"}
Nov 01 22:29:44.596292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.596260    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:44.716277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.716177    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\" (UniqueName: \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\") pod \"109bccba-5b18-412b-b6fd-09f4e49b64f6\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") "
Nov 01 22:29:44.716277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.716278    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\" (UniqueName: \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\") pod \"109bccba-5b18-412b-b6fd-09f4e49b64f6\" (UID: \"109bccba-5b18-412b-b6fd-09f4e49b64f6\") "
Nov 01 22:29:44.716631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.716603    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file" pod "109bccba-5b18-412b-b6fd-09f4e49b64f6" (UID: "109bccba-5b18-412b-b6fd-09f4e49b64f6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:44.718574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.718520    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj" pod "109bccba-5b18-412b-b6fd-09f4e49b64f6" (UID: "109bccba-5b18-412b-b6fd-09f4e49b64f6"). InnerVolumeSpecName "kube-api-access-t2ldj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:44.817617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.817569    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2ldj\" (UniqueName: \"kubernetes.io/projected/109bccba-5b18-412b-b6fd-09f4e49b64f6-kube-api-access-t2ldj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:44.817617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:44.817602    5031 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/109bccba-5b18-412b-b6fd-09f4e49b64f6-config-file\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:45.492692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:45.492656    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd" event={"ID":"109bccba-5b18-412b-b6fd-09f4e49b64f6","Type":"ContainerDied","Data":"73b2b2eb9933ac57f1682d140f04f27a13c40637a346d1e1c2f1a95ba5b91598"}
Nov 01 22:29:45.493100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:45.492699    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"
Nov 01 22:29:45.493100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:45.492730    5031 scope.go:122] "RemoveContainer" containerID="511c8994d623cf1192ce3bb6da4438448944bf91b55aedf5d2a1e4471bb68823"
Nov 01 22:29:45.647506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:45.647469    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:45.693490 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:45.693458    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kv8hd"]
Nov 01 22:29:46.168911 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.168877    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109bccba-5b18-412b-b6fd-09f4e49b64f6" path="/var/lib/kubelet/pods/109bccba-5b18-412b-b6fd-09f4e49b64f6/volumes"
Nov 01 22:29:46.249959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.249927    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"]
Nov 01 22:29:46.250254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.250240    5031 cpu_manager.go:414] "containerMap: removing container" podUID="109bccba-5b18-412b-b6fd-09f4e49b64f6" containerName="limitador"
Nov 01 22:29:46.250295 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.250256    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109bccba-5b18-412b-b6fd-09f4e49b64f6" containerName="limitador"
Nov 01 22:29:46.250329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.250323    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="109bccba-5b18-412b-b6fd-09f4e49b64f6" containerName="limitador"
Nov 01 22:29:46.252916 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.252894    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.331158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.331119    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-cert\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.331342 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.331213    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kj6\" (UniqueName: \"kubernetes.io/projected/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-kube-api-access-d4kj6\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.331342 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.331262    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-manager-config\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.331454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.331376    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-metrics-certs\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.347967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.347925    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-q64lq\""
Nov 01 22:29:46.347967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.347925    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 22:29:46.347967 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.347957    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 22:29:46.348195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.347977    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 22:29:46.348989 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.348968    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"]
Nov 01 22:29:46.432646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.432527    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-cert\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.432646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.432597    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kj6\" (UniqueName: \"kubernetes.io/projected/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-kube-api-access-d4kj6\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.432646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.432628    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-manager-config\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.432945 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.432671    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-metrics-certs\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.433478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.433420    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-manager-config\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.435312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.435262    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-metrics-certs\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.435457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.435357    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-cert\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.490706 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.490671    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kj6\" (UniqueName: \"kubernetes.io/projected/6cd817e8-65df-4b44-8a01-b3e3ee02c6c5-kube-api-access-d4kj6\") pod \"jobset-controller-manager-6445d5c664-m7dzr\" (UID: \"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5\") " pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.564650 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.564606    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:46.932982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:46.932942    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"]
Nov 01 22:29:46.939300 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:29:46.939264    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd817e8_65df_4b44_8a01_b3e3ee02c6c5.slice/crio-a2b421b18bb3f1d9190b40ee0140a1df3931566c5f95022f770466a78f6cb6ea WatchSource:0}: Error finding container a2b421b18bb3f1d9190b40ee0140a1df3931566c5f95022f770466a78f6cb6ea: Status 404 returned error can't find the container with id a2b421b18bb3f1d9190b40ee0140a1df3931566c5f95022f770466a78f6cb6ea
Nov 01 22:29:47.503630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:47.503591    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr" event={"ID":"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5","Type":"ContainerStarted","Data":"a2b421b18bb3f1d9190b40ee0140a1df3931566c5f95022f770466a78f6cb6ea"}
Nov 01 22:29:52.527058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:52.527020    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr" event={"ID":"6cd817e8-65df-4b44-8a01-b3e3ee02c6c5","Type":"ContainerStarted","Data":"a736a1362584699225efbdd70d277976b608b1b1120362e2a7e5576f2002f857"}
Nov 01 22:29:52.527646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:52.527125    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:29:52.638117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:29:52.638062    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr" podStartSLOduration=1.6278494700000001 podStartE2EDuration="6.638046261s" totalImagesPullingTime="5.010196791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:52.635649617 +0000 UTC m=+346.591689465" watchObservedRunningTime="2026-11-01 22:29:52.638046261 +0000 UTC m=+346.594086105"
Nov 01 22:30:03.537119 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:03.537082    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6445d5c664-m7dzr"
Nov 01 22:30:10.626413 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:30:10.626372    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:30:40.625123 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:30:40.625035    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:30:53.448020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.447975    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9lxd2"]
Nov 01 22:30:53.451458 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.451437    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.531737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.531684    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98rf\" (UniqueName: \"kubernetes.io/projected/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-kube-api-access-t98rf\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.531941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.531890    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-tls-certs\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.546560 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.546516    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 22:30:53.546560 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.546516    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 22:30:53.546749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.546567    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 22:30:53.546749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.546580    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rn7ql\""
Nov 01 22:30:53.547708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.547690    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9lxd2"]
Nov 01 22:30:53.633362 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.633316    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98rf\" (UniqueName: \"kubernetes.io/projected/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-kube-api-access-t98rf\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.633587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.633417    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-tls-certs\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.636067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.636038    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-tls-certs\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.686841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.686813    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98rf\" (UniqueName: \"kubernetes.io/projected/cfeef97a-4b32-4466-9af5-7f1977e4fa8b-kube-api-access-t98rf\") pod \"model-serving-api-75bd89fdfc-9lxd2\" (UID: \"cfeef97a-4b32-4466-9af5-7f1977e4fa8b\") " pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:53.763890 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:53.763798    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:54.133930 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:54.133894    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9lxd2"]
Nov 01 22:30:54.140300 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:30:54.140265    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeef97a_4b32_4466_9af5_7f1977e4fa8b.slice/crio-b59cbd8bf9fbb632b33644ad8da9b1686e105cddb40ed7c7dcc71a82d6912068 WatchSource:0}: Error finding container b59cbd8bf9fbb632b33644ad8da9b1686e105cddb40ed7c7dcc71a82d6912068: Status 404 returned error can't find the container with id b59cbd8bf9fbb632b33644ad8da9b1686e105cddb40ed7c7dcc71a82d6912068
Nov 01 22:30:54.777619 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:54.777580    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2" event={"ID":"cfeef97a-4b32-4466-9af5-7f1977e4fa8b","Type":"ContainerStarted","Data":"b59cbd8bf9fbb632b33644ad8da9b1686e105cddb40ed7c7dcc71a82d6912068"}
Nov 01 22:30:56.789046 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:56.789009    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2" event={"ID":"cfeef97a-4b32-4466-9af5-7f1977e4fa8b","Type":"ContainerStarted","Data":"a2a55473d9c8a96e8bc78866cb309e955cbb38b9b9249b03ea06239d69f41b47"}
Nov 01 22:30:56.789489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:56.789132    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:30:56.895573 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:30:56.895451    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2" podStartSLOduration=1.457836587 podStartE2EDuration="3.89543729s" totalImagesPullingTime="2.437600703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:56.894680459 +0000 UTC m=+410.850720303" watchObservedRunningTime="2026-11-01 22:30:56.89543729 +0000 UTC m=+410.851477127"
Nov 01 22:31:00.724982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.724948    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 22:31:00.728430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.728409    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.777419 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.777384    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 22:31:00.824232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.824197    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 22:31:00.824435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.824265    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 22:31:00.824435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.824315    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 22:31:00.824435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.824383    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 22:31:00.825385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.825364    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 22:31:00.897503 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.897467    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.897730 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.897519    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.897730 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.897653    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.897839 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.897739    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm22\" (UniqueName: \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.998281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.998198    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.998281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.998255    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.998281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.998282    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm22\" (UniqueName: \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.998495 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.998375    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:00.999112 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:00.999018    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:01.000972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.000949    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:01.001094 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.001072    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:01.052568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.052517    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm22\" (UniqueName: \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\") pod \"maas-api-5bfbcdfbbb-2nn64\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") " pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:01.341148 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.341116    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:01.707637 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.707555    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 22:31:01.714254 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:31:01.714226    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adb6621_c7c7_42be_be85_804dc8a64f04.slice/crio-f7cb9109800dc9003b9288b7c040a5c57062fd97c95e1343bbcf48b0a18e95b2 WatchSource:0}: Error finding container f7cb9109800dc9003b9288b7c040a5c57062fd97c95e1343bbcf48b0a18e95b2: Status 404 returned error can't find the container with id f7cb9109800dc9003b9288b7c040a5c57062fd97c95e1343bbcf48b0a18e95b2
Nov 01 22:31:01.810233 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:01.810197    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" event={"ID":"0adb6621-c7c7-42be-be85-804dc8a64f04","Type":"ContainerStarted","Data":"f7cb9109800dc9003b9288b7c040a5c57062fd97c95e1343bbcf48b0a18e95b2"}
Nov 01 22:31:04.824813 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:04.824778    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" event={"ID":"0adb6621-c7c7-42be-be85-804dc8a64f04","Type":"ContainerStarted","Data":"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"}
Nov 01 22:31:04.825232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:04.824892    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:04.941122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:04.941071    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" podStartSLOduration=1.925709334 podStartE2EDuration="4.94105193s" totalImagesPullingTime="3.015342596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:04.940437831 +0000 UTC m=+418.896477676" watchObservedRunningTime="2026-11-01 22:31:04.94105193 +0000 UTC m=+418.897091775"
Nov 01 22:31:07.797649 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:07.797616    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-9lxd2"
Nov 01 22:31:10.626595 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:31:10.626556    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:31:10.835488 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:31:10.835456    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 22:31:40.626326 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:31:40.626285    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:32:10.625681 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:32:10.625645    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:32:40.625548 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:32:40.625491    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:33:02.918017 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:02.917980    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"]
Nov 01 22:33:02.921307 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:02.921263    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:02.973675 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:02.973643    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 22:33:03.017872 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.017837    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80903051-820f-4d46-ac36-c364a4f19caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.017890    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.017974    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frfxq\" (UniqueName: \"kubernetes.io/projected/80903051-820f-4d46-ac36-c364a4f19caf-kube-api-access-frfxq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.018029    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018142 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.018069    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018142 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.018085    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.018977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.018957    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 22:33:03.019068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.019021    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 22:33:03.019113 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.019101    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4662b\""
Nov 01 22:33:03.020718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.020696    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"]
Nov 01 22:33:03.118563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.118563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118560    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frfxq\" (UniqueName: \"kubernetes.io/projected/80903051-820f-4d46-ac36-c364a4f19caf-kube-api-access-frfxq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.118810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118596    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.118810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118628    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.118810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118643    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.118810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.118709    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80903051-820f-4d46-ac36-c364a4f19caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.119063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.119037    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.119171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.119077    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.119171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.119117    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.121011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.120983    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80903051-820f-4d46-ac36-c364a4f19caf-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.121239 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.121223    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80903051-820f-4d46-ac36-c364a4f19caf-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.172262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.172180    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frfxq\" (UniqueName: \"kubernetes.io/projected/80903051-820f-4d46-ac36-c364a4f19caf-kube-api-access-frfxq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vws56\" (UID: \"80903051-820f-4d46-ac36-c364a4f19caf\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.234655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.234613    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:03.600706 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:03.600665    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"]
Nov 01 22:33:03.607865 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:03.607828    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80903051_820f_4d46_ac36_c364a4f19caf.slice/crio-94561d6eb01bd8898a6fbd9a4d909fd7ff35cf9acf22f257199022e9febe47aa WatchSource:0}: Error finding container 94561d6eb01bd8898a6fbd9a4d909fd7ff35cf9acf22f257199022e9febe47aa: Status 404 returned error can't find the container with id 94561d6eb01bd8898a6fbd9a4d909fd7ff35cf9acf22f257199022e9febe47aa
Nov 01 22:33:04.324959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:04.324923    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56" event={"ID":"80903051-820f-4d46-ac36-c364a4f19caf","Type":"ContainerStarted","Data":"94561d6eb01bd8898a6fbd9a4d909fd7ff35cf9acf22f257199022e9febe47aa"}
Nov 01 22:33:05.330454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:05.330409    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56" event={"ID":"80903051-820f-4d46-ac36-c364a4f19caf","Type":"ContainerStarted","Data":"c2a3a230206eec9645bf9ad2aa673a66809d1abd7aa2a34b29443b645b4fea14"}
Nov 01 22:33:05.436453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:05.436397    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56" podStartSLOduration=1.841406551 podStartE2EDuration="3.436382634s" totalImagesPullingTime="1.594976083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:05.436241201 +0000 UTC m=+539.392281026" watchObservedRunningTime="2026-11-01 22:33:05.436382634 +0000 UTC m=+539.392422475"
Nov 01 22:33:06.235524 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:06.235489    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:09.488343 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.488313    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"]
Nov 01 22:33:09.491810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.491777    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576526 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576489    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576614    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576647    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4hx\" (UniqueName: \"kubernetes.io/projected/6344d70b-d83b-41e7-b1ef-51648a96835a-kube-api-access-ht4hx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576676    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576710    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.576756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.576737    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6344d70b-d83b-41e7-b1ef-51648a96835a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.586100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.586070    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 22:33:09.587467 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.587444    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"]
Nov 01 22:33:09.678086 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678048    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678094    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4hx\" (UniqueName: \"kubernetes.io/projected/6344d70b-d83b-41e7-b1ef-51648a96835a-kube-api-access-ht4hx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678143    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678175    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6344d70b-d83b-41e7-b1ef-51648a96835a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678338    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678496    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678636 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678582    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.678680 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.678635    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.680724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.680695    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6344d70b-d83b-41e7-b1ef-51648a96835a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.680861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.680826    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6344d70b-d83b-41e7-b1ef-51648a96835a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.734260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.734222    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4hx\" (UniqueName: \"kubernetes.io/projected/6344d70b-d83b-41e7-b1ef-51648a96835a-kube-api-access-ht4hx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c\" (UID: \"6344d70b-d83b-41e7-b1ef-51648a96835a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:09.804274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:09.804240    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:10.172820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:10.172731    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"]
Nov 01 22:33:10.180242 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:10.180201    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6344d70b_d83b_41e7_b1ef_51648a96835a.slice/crio-bca5de590b57490e736836e390ece57e4a0ed5296facfdc9033bb1c376ce6ba2 WatchSource:0}: Error finding container bca5de590b57490e736836e390ece57e4a0ed5296facfdc9033bb1c376ce6ba2: Status 404 returned error can't find the container with id bca5de590b57490e736836e390ece57e4a0ed5296facfdc9033bb1c376ce6ba2
Nov 01 22:33:10.351314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:10.351279    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c" event={"ID":"6344d70b-d83b-41e7-b1ef-51648a96835a","Type":"ContainerStarted","Data":"bca5de590b57490e736836e390ece57e4a0ed5296facfdc9033bb1c376ce6ba2"}
Nov 01 22:33:10.624697 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:33:10.624662    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:33:11.363246 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:11.363203    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c" event={"ID":"6344d70b-d83b-41e7-b1ef-51648a96835a","Type":"ContainerStarted","Data":"6db01c7b0c2f9207ffed69c9dbe274f339d10bbe820ed15686a781d7c62be701"}
Nov 01 22:33:11.470868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:11.470808    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c" podStartSLOduration=1.821846544 podStartE2EDuration="2.470794522s" totalImagesPullingTime="648.947978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:11.46950804 +0000 UTC m=+545.425547884" watchObservedRunningTime="2026-11-01 22:33:11.470794522 +0000 UTC m=+545.426834367"
Nov 01 22:33:11.804627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:11.804589    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:13.235612 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:13.235576    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:13.248201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:13.248171    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:13.383156 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:13.383129    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vws56"
Nov 01 22:33:19.805208 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:19.805104    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:19.818025 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:19.817992    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:20.415092 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.415059    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c"
Nov 01 22:33:20.775603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.775502    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"]
Nov 01 22:33:20.781652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.781622    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.832036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.832007    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 22:33:20.839282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.839253    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"]
Nov 01 22:33:20.876384 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876350    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.876384 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876395    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.876795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876511    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.876795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876620    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.876795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876643    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4w77\" (UniqueName: \"kubernetes.io/projected/0d80334c-9fa2-413b-94ea-a4c353344f29-kube-api-access-w4w77\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.876795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.876674    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80334c-9fa2-413b-94ea-a4c353344f29-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977599    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977648    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977675    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977695    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4w77\" (UniqueName: \"kubernetes.io/projected/0d80334c-9fa2-413b-94ea-a4c353344f29-kube-api-access-w4w77\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.977720    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80334c-9fa2-413b-94ea-a4c353344f29-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.978043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.978018    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.978104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.978073    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.978160 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.978141    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.980144 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.980117    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d80334c-9fa2-413b-94ea-a4c353344f29-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:20.980387 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:20.980370    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d80334c-9fa2-413b-94ea-a4c353344f29-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:21.033271 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:21.033194    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4w77\" (UniqueName: \"kubernetes.io/projected/0d80334c-9fa2-413b-94ea-a4c353344f29-kube-api-access-w4w77\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx\" (UID: \"0d80334c-9fa2-413b-94ea-a4c353344f29\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:21.094226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:21.094196    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:21.462991 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:21.462959    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"]
Nov 01 22:33:21.469578 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:21.469530    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d80334c_9fa2_413b_94ea_a4c353344f29.slice/crio-153c215003e53fae8d1a1732778b86ebf3db0d7ff1caeaf21533afbf2ad39b38 WatchSource:0}: Error finding container 153c215003e53fae8d1a1732778b86ebf3db0d7ff1caeaf21533afbf2ad39b38: Status 404 returned error can't find the container with id 153c215003e53fae8d1a1732778b86ebf3db0d7ff1caeaf21533afbf2ad39b38
Nov 01 22:33:22.415303 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:22.415198    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" event={"ID":"0d80334c-9fa2-413b-94ea-a4c353344f29","Type":"ContainerStarted","Data":"14ec6922b2f31d1f9360f8d3aedd772308deb42d6054d3ccbd407ac26a073e6d"}
Nov 01 22:33:22.415303 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:22.415250    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" event={"ID":"0d80334c-9fa2-413b-94ea-a4c353344f29","Type":"ContainerStarted","Data":"153c215003e53fae8d1a1732778b86ebf3db0d7ff1caeaf21533afbf2ad39b38"}
Nov 01 22:33:22.526308 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:22.526248    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podStartSLOduration=1.912172856 podStartE2EDuration="2.526230181s" totalImagesPullingTime="614.057325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:22.525343169 +0000 UTC m=+556.481383059" watchObservedRunningTime="2026-11-01 22:33:22.526230181 +0000 UTC m=+556.482270024"
Nov 01 22:33:23.095243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.095209    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:23.592448 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.592413    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"]
Nov 01 22:33:23.596071 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.596048    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.644604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.644533    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 22:33:23.695104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.695063    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"]
Nov 01 22:33:23.702398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702361    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702402    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702563    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702599    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702627    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702722    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702750    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702915 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.702973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.702919    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85z2\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-kube-api-access-r85z2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.739257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.739220    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv4kj\""
Nov 01 22:33:23.803783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803740    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.803783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803784    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803806    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803875    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803896    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803917    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.803996    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85z2\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-kube-api-access-r85z2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804031    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804237    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804317    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804673    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.804989 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.804965    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.806361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.806325    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.806520 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.806502    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.857505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.857429    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.857661 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.857585    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85z2\" (UniqueName: \"kubernetes.io/projected/e35dfd80-6c4d-47df-9581-698a23fe2ed3-kube-api-access-r85z2\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fx6mf\" (UID: \"e35dfd80-6c4d-47df-9581-698a23fe2ed3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:23.908403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:23.908365    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:24.328130 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:24.328098    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"]
Nov 01 22:33:24.334576 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:24.334520    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35dfd80_6c4d_47df_9581_698a23fe2ed3.slice/crio-2b4a18b8caf04603735802345eaae348472516bf76e4e5b3a7ec869d3d8e38a7 WatchSource:0}: Error finding container 2b4a18b8caf04603735802345eaae348472516bf76e4e5b3a7ec869d3d8e38a7: Status 404 returned error can't find the container with id 2b4a18b8caf04603735802345eaae348472516bf76e4e5b3a7ec869d3d8e38a7
Nov 01 22:33:24.426852 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:24.426818    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" event={"ID":"e35dfd80-6c4d-47df-9581-698a23fe2ed3","Type":"ContainerStarted","Data":"2b4a18b8caf04603735802345eaae348472516bf76e4e5b3a7ec869d3d8e38a7"}
Nov 01 22:33:27.191482 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.191439    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"]
Nov 01 22:33:27.195186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.195160    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.289813 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.289776    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 01 22:33:27.290988 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.290966    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"]
Nov 01 22:33:27.337460 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337425    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.337654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337597    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a603f6-220c-44c4-aae2-13cd972b0187-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.337712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337684    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.337712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337709    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.337794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337756    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.337794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.337781    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlgj\" (UniqueName: \"kubernetes.io/projected/d3a603f6-220c-44c4-aae2-13cd972b0187-kube-api-access-2tlgj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438437    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438492    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438511    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438551    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlgj\" (UniqueName: \"kubernetes.io/projected/d3a603f6-220c-44c4-aae2-13cd972b0187-kube-api-access-2tlgj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438611    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438668    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a603f6-220c-44c4-aae2-13cd972b0187-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438907    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.438969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.438943    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.439260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.439226    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.441320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.441261    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a603f6-220c-44c4-aae2-13cd972b0187-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.441934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.441874    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a603f6-220c-44c4-aae2-13cd972b0187-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.491813 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.491779    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlgj\" (UniqueName: \"kubernetes.io/projected/d3a603f6-220c-44c4-aae2-13cd972b0187-kube-api-access-2tlgj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-vjfhb\" (UID: \"d3a603f6-220c-44c4-aae2-13cd972b0187\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.508225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.508193    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:27.884950 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:27.884916    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"]
Nov 01 22:33:27.891508 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:27.891476    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a603f6_220c_44c4_aae2_13cd972b0187.slice/crio-8a1bb16da99fd7e8c18b2fd3695d43a42e970c29317db18698c6de3447164256 WatchSource:0}: Error finding container 8a1bb16da99fd7e8c18b2fd3695d43a42e970c29317db18698c6de3447164256: Status 404 returned error can't find the container with id 8a1bb16da99fd7e8c18b2fd3695d43a42e970c29317db18698c6de3447164256
Nov 01 22:33:28.448260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:28.448223    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb" event={"ID":"d3a603f6-220c-44c4-aae2-13cd972b0187","Type":"ContainerStarted","Data":"8a1bb16da99fd7e8c18b2fd3695d43a42e970c29317db18698c6de3447164256"}
Nov 01 22:33:28.719126 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:28.719083    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:28.719230 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:28.719162    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:28.719230 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:28.719193    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:29.479432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.479391    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" event={"ID":"e35dfd80-6c4d-47df-9581-698a23fe2ed3","Type":"ContainerStarted","Data":"a088b8cbbe9bcda3c32be68ef11508ae777d82fb0798eab655a4f5a4f3dc208c"}
Nov 01 22:33:29.481221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.481189    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb" event={"ID":"d3a603f6-220c-44c4-aae2-13cd972b0187","Type":"ContainerStarted","Data":"6c6c5009023cf6df6ef952effb381f7bc70e473829a8df61c0199825bd16599a"}
Nov 01 22:33:29.509169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.509132    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:29.637506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.637448    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podStartSLOduration=2.25601794 podStartE2EDuration="6.637431752s" totalImagesPullingTime="4.381413812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:29.63595062 +0000 UTC m=+563.591990464" watchObservedRunningTime="2026-11-01 22:33:29.637431752 +0000 UTC m=+563.593471656"
Nov 01 22:33:29.749753 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.749646    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb" podStartSLOduration=2.151157138 podStartE2EDuration="2.749630012s" totalImagesPullingTime="598.472874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:29.747909403 +0000 UTC m=+563.703949247" watchObservedRunningTime="2026-11-01 22:33:29.749630012 +0000 UTC m=+563.705669855"
Nov 01 22:33:29.908827 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.908784    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:29.910403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.910374    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:29.910515 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:29.910447    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:30.909378 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:30.909340    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:30.909848 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:30.909399    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:31.094882 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:31.094837    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:31.107576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:31.107526    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:31.501391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:31.501359    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:33:31.908959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:31.908917    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:31.909140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:31.908985    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:32.909201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:32.909159    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:32.909663 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:32.909225    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:33.909069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:33.909034    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:33.909262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:33.909090    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:34.909495 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:34.909448    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:34.910018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:34.909521    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:35.908775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:35.908740    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body=
Nov 01 22:33:35.908955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:35.908799    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused"
Nov 01 22:33:37.508452 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:37.508414    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:37.526598 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:37.526562    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:37.537783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:37.537755    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-vjfhb"
Nov 01 22:33:37.950763 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:37.950717    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:33:37.950951 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:37.950775    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:33:38.568849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.568820    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:38.569393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.569120    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:38.569821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.569805    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf"
Nov 01 22:33:38.786645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.786614    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 22:33:38.789928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.789906    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862593 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862485    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862593 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862524    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862638    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862678    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862807    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.862874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862838    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.863034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862905    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.863034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.862960    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc5x\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.863101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.863038    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.880909 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.880861    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 22:33:38.964188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964151    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc5x\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964205    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964250    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964311    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964339    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964414    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964439    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.964715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964473    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.965474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.964820    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.965474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.965403    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.965474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.965463    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.965663 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.965514    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.965700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.965662    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.967397 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.967369    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:38.967620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:38.967605    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:39.019783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.019752    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:39.020046 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.020024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc5x\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\") pod \"maas-default-gateway-openshift-default-7f694df8bc-xvntw\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:39.102124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.102082    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:39.526140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.526060    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 22:33:39.531662 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:33:39.531625    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21812fc_97a7_4b8d_9f2e_ecb49e5a7e49.slice/crio-0e96df5a471ad974bb2f05a339329cd192244ac0d4ab06875fde3fdd5c76de62 WatchSource:0}: Error finding container 0e96df5a471ad974bb2f05a339329cd192244ac0d4ab06875fde3fdd5c76de62: Status 404 returned error can't find the container with id 0e96df5a471ad974bb2f05a339329cd192244ac0d4ab06875fde3fdd5c76de62
Nov 01 22:33:39.533614 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.533571    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:39.533716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.533655    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:39.533716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:39.533696    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:33:40.536438 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:40.536397    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" event={"ID":"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49","Type":"ContainerStarted","Data":"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"}
Nov 01 22:33:40.536955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:40.536459    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" event={"ID":"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49","Type":"ContainerStarted","Data":"0e96df5a471ad974bb2f05a339329cd192244ac0d4ab06875fde3fdd5c76de62"}
Nov 01 22:33:40.625122 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:33:40.625086    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:33:40.646260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:40.646201    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podStartSLOduration=2.646182288 podStartE2EDuration="2.646182288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:40.645242685 +0000 UTC m=+574.601282529" watchObservedRunningTime="2026-11-01 22:33:40.646182288 +0000 UTC m=+574.602222133"
Nov 01 22:33:41.102522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:41.102485    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:41.104072 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:41.104051    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:41.104144 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:41.104126    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:42.103403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:42.103366    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:42.103943 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:42.103430    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:43.102801 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:43.102765    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:43.102977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:43.102824    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:44.102716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:44.102679    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:44.103132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:44.102756    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:45.102742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:45.102705    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:45.103181 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:45.102759    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:46.102955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:46.102915    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 22:33:46.103418 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:46.102972    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 22:33:47.911733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:47.911647    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:47.912236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:47.911993    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:33:47.912940 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:33:47.912920    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 22:34:10.625931 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:34:10.625888    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:34:24.068258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.068224    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:34:24.071708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.071686    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.092832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.092797    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.092832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.092829    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.092850    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.092960    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.093014    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093125 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.093085    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hqm\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093125 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.093110    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.093128    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.093186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.093145    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.173824 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.173775    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:34:24.193756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.193717    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hqm\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.193954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.193763    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.193954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.193940    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194077 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.193989    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194107    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194169    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194239    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194265    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194417    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.194894 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194842    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.195005 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194894    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.195005 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.194943    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.195087 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.195024    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.196298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.196278    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.196746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.196726    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.248236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.248203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hqm\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.248417 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.248291    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hbrrp\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.384548 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.384451    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:24.809491 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.809452    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:34:24.815882 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:34:24.815846    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b26e64_e1e1_4e85_8ab9_abac4ddb2f08.slice/crio-5ef214b065d2c7f780f3c669429cef5e53692f594e9ae068a4870b265930c262 WatchSource:0}: Error finding container 5ef214b065d2c7f780f3c669429cef5e53692f594e9ae068a4870b265930c262: Status 404 returned error can't find the container with id 5ef214b065d2c7f780f3c669429cef5e53692f594e9ae068a4870b265930c262
Nov 01 22:34:24.817959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.817914    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:24.818084 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.818064    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:24.818156 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:24.818112    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:25.738474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:25.738439    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" event={"ID":"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08","Type":"ContainerStarted","Data":"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"}
Nov 01 22:34:25.739007 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:25.738489    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" event={"ID":"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08","Type":"ContainerStarted","Data":"5ef214b065d2c7f780f3c669429cef5e53692f594e9ae068a4870b265930c262"}
Nov 01 22:34:25.901736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:25.900693    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podStartSLOduration=1.90067359 podStartE2EDuration="1.90067359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:25.898086442 +0000 UTC m=+619.854126278" watchObservedRunningTime="2026-11-01 22:34:25.90067359 +0000 UTC m=+619.856713438"
Nov 01 22:34:26.384822 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:26.384778    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:26.386469 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:26.386429    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:26.386614 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:26.386521    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:27.385145 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:27.385103    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:27.385576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:27.385175    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:28.385016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:28.384982    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:28.385203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:28.385044    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:29.385041 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:29.385002    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:29.385257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:29.385066    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:30.385076 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:30.385040    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:30.385268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:30.385091    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:31.384976 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:31.384936    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:31.385169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:31.385023    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:32.385818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:32.385780    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hbrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 22:34:32.386306 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:32.385849    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 22:34:33.389427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:33.389392    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:33.389914 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:33.389781    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:33.390502 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:33.390483    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:34:40.625228 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:34:40.625187    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:34:52.994134 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.994051    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:34:52.997674 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.997644    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998135    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998176    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998317 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998264    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998356 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998320    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998422 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998402    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998439    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998504    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998524    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:52.998646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:52.998604    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6jv\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.056257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.056212    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b8fd2-openshift-default-dockercfg-bqfks\""
Nov 01 22:34:53.099505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099472    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099505    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099529    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099579    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099604    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099645    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6jv\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099696    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099727    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.099775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099769    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.100188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.099983    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.100188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.100011    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.100188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.100125    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.100315 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.100255    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.100387 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.100363    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.102152 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.102130    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.102286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.102269    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.119253 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.119217    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:34:53.155676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.155630    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6jv\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.159375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.159348    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.312193 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.312165    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:53.499497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.499450    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 22:34:53.505181 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.505152    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.560801 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.560769    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-d6101b-openshift-default-dockercfg-jbnc7\""
Nov 01 22:34:53.574381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.574310    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 22:34:53.603724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.603675    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.603932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.603732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.603932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.603803    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.603932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.603911    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.604124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.603976    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.604124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.604087    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.604232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.604135    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.604286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.604254    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5s2\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.604529 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.604284    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.693012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.692979    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:34:53.699469 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:34:53.699438    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3a252d_49cf_415c_8a86_80e048220cd4.slice/crio-4a7235e21ef9d00a24021ca5fe6095f0ccfb79c992797c02fdb491ad7f294f6f WatchSource:0}: Error finding container 4a7235e21ef9d00a24021ca5fe6095f0ccfb79c992797c02fdb491ad7f294f6f: Status 404 returned error can't find the container with id 4a7235e21ef9d00a24021ca5fe6095f0ccfb79c992797c02fdb491ad7f294f6f
Nov 01 22:34:53.701632 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.701600    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:53.701710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.701670    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:53.701710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.701701    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:53.704797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704775    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5s2\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704810    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704851    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704997 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704885    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704997 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704997 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704943    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.704997 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.704977    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705050    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705097    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705200    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705520    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705877 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705852    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.705959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705915    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.706018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.705998    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.707751 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.707721    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.707941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.707925    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.761437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.761404    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.761966 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.761944    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5s2\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.822610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.822573    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:53.893651 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.892665    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerStarted","Data":"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"}
Nov 01 22:34:53.893651 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:53.892717    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerStarted","Data":"4a7235e21ef9d00a24021ca5fe6095f0ccfb79c992797c02fdb491ad7f294f6f"}
Nov 01 22:34:54.072396 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.072333    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podStartSLOduration=2.072316645 podStartE2EDuration="2.072316645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.071331937 +0000 UTC m=+648.027371783" watchObservedRunningTime="2026-11-01 22:34:54.072316645 +0000 UTC m=+648.028356488"
Nov 01 22:34:54.207661 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.207567    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 22:34:54.215342 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:34:54.215306    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b11edd3_a9a7_4607_bd7a_9615671ef7df.slice/crio-d4371ccd7243a7ce92e29d0bc41657cfb3b57bedfed4e8d4ea6465ed272fb7ce WatchSource:0}: Error finding container d4371ccd7243a7ce92e29d0bc41657cfb3b57bedfed4e8d4ea6465ed272fb7ce: Status 404 returned error can't find the container with id d4371ccd7243a7ce92e29d0bc41657cfb3b57bedfed4e8d4ea6465ed272fb7ce
Nov 01 22:34:54.217642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.217601    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:54.217768 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.217671    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:54.217768 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.217724    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:54.312376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.312352    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:54.317672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.317641    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:54.539551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.539450    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 22:34:54.543178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.543152    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.609399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.609360    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a6dd2f-openshift-default-dockercfg-vvpcs\""
Nov 01 22:34:54.612119 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.612089    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 22:34:54.615600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615526    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.615600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615590    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.615846 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615714    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.615846 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.615955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615875    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.615955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615936    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.616057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.615984    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.616114 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.616058    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.616114 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.616087    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99db8\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717426    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717479    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717508    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717529    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717564    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99db8\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717617    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.717671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717651    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717700    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.717736    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718196 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.718159    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718301 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.718281    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.718314    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.718349    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.718441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.718382    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.720238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.720217    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.720615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.720592    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.772054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.772017    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.772223 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.772158    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99db8\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.857022 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.856992    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:54.881878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.881845    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 22:34:54.885688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.885658    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.900117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.900082    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerStarted","Data":"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"}
Nov 01 22:34:54.900293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.900148    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerStarted","Data":"d4371ccd7243a7ce92e29d0bc41657cfb3b57bedfed4e8d4ea6465ed272fb7ce"}
Nov 01 22:34:54.900293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.900177    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:54.903085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.903054    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:34:54.920258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920220    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920287    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920443    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl87z\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920507    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920584    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920628    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920768 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920708    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.920768 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.920749    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:54.936810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.936478    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c768f3-openshift-default-dockercfg-r664d\""
Nov 01 22:34:54.952519 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:54.952488    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 22:34:55.021532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021496    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021552    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021580    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021597    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl87z\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021617    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021640    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021658    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021708    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.021794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.021748    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.022161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.022056    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.022161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.022085    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.022238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.022157    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.022288 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.022274    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.022845 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.022820    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.024057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.024037    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.024290 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.024271    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.075354 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.075320    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.075772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.075567    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl87z\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.112184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.112074    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podStartSLOduration=2.112059173 podStartE2EDuration="2.112059173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:55.111128979 +0000 UTC m=+649.067168823" watchObservedRunningTime="2026-11-01 22:34:55.112059173 +0000 UTC m=+649.068099017"
Nov 01 22:34:55.200255 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.200221    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:55.251565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.248111    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 22:34:55.280980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.280704    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.280980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.280798    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.280980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.280843    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.597733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.597699    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 22:34:55.604493 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:34:55.604457    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a203e67_a35d_4644_a6ac_ba1d1de22534.slice/crio-fb5933304b816f57085dfd438163020335b1026e81116e78bd585fe38ec53e73 WatchSource:0}: Error finding container fb5933304b816f57085dfd438163020335b1026e81116e78bd585fe38ec53e73: Status 404 returned error can't find the container with id fb5933304b816f57085dfd438163020335b1026e81116e78bd585fe38ec53e73
Nov 01 22:34:55.606962 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.606926    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.607056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.606997    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.607056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.607028    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:34:55.823846 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.823804    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:55.829133 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.829103    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:55.906255 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.906149    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerStarted","Data":"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"}
Nov 01 22:34:55.906255 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.906219    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerStarted","Data":"fb5933304b816f57085dfd438163020335b1026e81116e78bd585fe38ec53e73"}
Nov 01 22:34:55.907880 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.907844    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerStarted","Data":"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"}
Nov 01 22:34:55.908031 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.907893    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerStarted","Data":"06c9bd5d883d3f67e55728d6e9ac45b7966f2544d543da3f7f7c8158f77df99d"}
Nov 01 22:34:55.908259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.908242    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:55.909282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:55.909261    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:34:56.197936 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.197838    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podStartSLOduration=2.19782253 podStartE2EDuration="2.19782253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:56.196533258 +0000 UTC m=+650.152573102" watchObservedRunningTime="2026-11-01 22:34:56.19782253 +0000 UTC m=+650.153862373"
Nov 01 22:34:56.200774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.200748    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:56.206615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.206587    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:56.328980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.328921    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podStartSLOduration=2.328901199 podStartE2EDuration="2.328901199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:56.326360171 +0000 UTC m=+650.282400016" watchObservedRunningTime="2026-11-01 22:34:56.328901199 +0000 UTC m=+650.284941047"
Nov 01 22:34:56.857687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.857651    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:56.862942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.862911    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:56.912600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.912571    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:34:56.912756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.912623    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:56.913188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.913168    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:34:56.913292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:34:56.913238    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:35:03.647812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.647782    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 22:35:03.652660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.652637    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.702028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.701993    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5b8fd2-serving-cert\""
Nov 01 22:35:03.715705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.715679    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.715816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.715742    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcbv\" (UniqueName: \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.715862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.715821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.715910 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.715893    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.750134 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.750100    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 22:35:03.817406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.817360    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.817629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.817447    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.817629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.817499    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcbv\" (UniqueName: \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.817629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.817576    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.817810 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:03.817627    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-serving-cert: secret "maas-api-e2e-worker-w1-5b8fd2-serving-cert" not found
Nov 01 22:35:03.817810 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:03.817703    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls podName:cac4d6d1-1ede-40ae-b07b-ec3681fe9020 nodeName:}" failed. No retries permitted until 2026-11-01 22:35:04.31768592 +0000 UTC m=+658.273725745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls") pod "maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" (UID: "cac4d6d1-1ede-40ae-b07b-ec3681fe9020") : secret "maas-api-e2e-worker-w1-5b8fd2-serving-cert" not found
Nov 01 22:35:03.818141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.818122    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.820278 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.820251    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:03.874519 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:03.874478    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcbv\" (UniqueName: \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:04.324486 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:04.324440    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:04.327041 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:04.327010    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:04.566439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:04.566400    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:04.949577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:04.949472    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 22:35:04.956045 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:35:04.956007    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac4d6d1_1ede_40ae_b07b_ec3681fe9020.slice/crio-71a05181d0c0be518df5dfe6f75ee9fd1839d102a02a07e813bdf08d62438e2c WatchSource:0}: Error finding container 71a05181d0c0be518df5dfe6f75ee9fd1839d102a02a07e813bdf08d62438e2c: Status 404 returned error can't find the container with id 71a05181d0c0be518df5dfe6f75ee9fd1839d102a02a07e813bdf08d62438e2c
Nov 01 22:35:04.957384 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:04.957359    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:35:05.955165 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:05.955129    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" event={"ID":"cac4d6d1-1ede-40ae-b07b-ec3681fe9020","Type":"ContainerStarted","Data":"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"}
Nov 01 22:35:05.955574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:05.955187    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" event={"ID":"cac4d6d1-1ede-40ae-b07b-ec3681fe9020","Type":"ContainerStarted","Data":"71a05181d0c0be518df5dfe6f75ee9fd1839d102a02a07e813bdf08d62438e2c"}
Nov 01 22:35:05.955574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:05.955215    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:06.082455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:06.082344    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" podStartSLOduration=2.24020218 podStartE2EDuration="3.08232902s" totalImagesPullingTime="842.12684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:06.080908678 +0000 UTC m=+660.036948511" watchObservedRunningTime="2026-11-01 22:35:06.08232902 +0000 UTC m=+660.038368903"
Nov 01 22:35:10.627211 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:10.627165    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:35:11.966384 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:11.966355    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 22:35:14.848531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.848489    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 22:35:14.853507 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.853480    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:14.946127 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.946094    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:14.946326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.946162    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:14.946326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.946243    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9q5w\" (UniqueName: \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:14.946326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.946285    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:14.977565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.967752    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-e71d55-serving-cert\""
Nov 01 22:35:14.977565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:14.971304    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 22:35:15.047720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.047680    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.047889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.047741    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9q5w\" (UniqueName: \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.047889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.047788    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.048010 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.047925    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.048118 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:15.048095    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-serving-cert: secret "maas-api-e2e-worker-w3-e71d55-serving-cert" not found
Nov 01 22:35:15.048196 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:15.048185    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls podName:43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15 nodeName:}" failed. No retries permitted until 2026-11-01 22:35:15.548163196 +0000 UTC m=+669.504203020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls") pod "maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" (UID: "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15") : secret "maas-api-e2e-worker-w3-e71d55-serving-cert" not found
Nov 01 22:35:15.048559 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.048520    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.050391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.050372    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.107375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.107221    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9q5w\" (UniqueName: \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.552702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.552670    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.579480 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.579444    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:15.791964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:15.791917    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:16.196207 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:16.196121    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 22:35:16.202003 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:35:16.201971    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a3621a_b3bf_4ed8_a4f6_caf25dd8bc15.slice/crio-30d13d405b454120593c7a978044123e4dc2a0e9c65c589fab4df44c1b176c6d WatchSource:0}: Error finding container 30d13d405b454120593c7a978044123e4dc2a0e9c65c589fab4df44c1b176c6d: Status 404 returned error can't find the container with id 30d13d405b454120593c7a978044123e4dc2a0e9c65c589fab4df44c1b176c6d
Nov 01 22:35:17.019505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:17.019471    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" event={"ID":"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15","Type":"ContainerStarted","Data":"30d13d405b454120593c7a978044123e4dc2a0e9c65c589fab4df44c1b176c6d"}
Nov 01 22:35:18.026310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:18.026276    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" event={"ID":"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15","Type":"ContainerStarted","Data":"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"}
Nov 01 22:35:18.026786 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:18.026391    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:18.147238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:18.147181    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" podStartSLOduration=3.047948672 podStartE2EDuration="4.147166565s" totalImagesPullingTime="1.099217893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:18.146951682 +0000 UTC m=+672.102991525" watchObservedRunningTime="2026-11-01 22:35:18.147166565 +0000 UTC m=+672.103206409"
Nov 01 22:35:20.694605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.694564    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 22:35:20.698420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.698395    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.747139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.747108    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-d6101b-serving-cert\""
Nov 01 22:35:20.795055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.795006    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 22:35:20.835586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.835511    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.835800 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.835636    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.835800 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.835708    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwmh\" (UniqueName: \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.835921 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.835863    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.936930 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.936886    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwmh\" (UniqueName: \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.937124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.937007    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.937124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.937106    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.937248 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.937135    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.937700 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:20.937672    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-serving-cert: secret "maas-api-e2e-worker-w5-d6101b-serving-cert" not found
Nov 01 22:35:20.937874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.937742    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.937874 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:20.937752    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls podName:d8a585c4-897e-41d0-b122-19f8b56682e0 nodeName:}" failed. No retries permitted until 2026-11-01 22:35:21.437730101 +0000 UTC m=+675.393769946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls") pod "maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" (UID: "d8a585c4-897e-41d0-b122-19f8b56682e0") : secret "maas-api-e2e-worker-w5-d6101b-serving-cert" not found
Nov 01 22:35:20.958284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.958170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:20.992417 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:20.992380    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwmh\" (UniqueName: \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:21.442666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:21.442630    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:21.445326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:21.445293    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") pod \"maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:21.615235 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:21.615193    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:22.016147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.016059    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 22:35:22.022649 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:35:22.022613    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a585c4_897e_41d0_b122_19f8b56682e0.slice/crio-21461e6cebbe69398713aeb0798997b88157bd0357c11eadc984855366f1d6ad WatchSource:0}: Error finding container 21461e6cebbe69398713aeb0798997b88157bd0357c11eadc984855366f1d6ad: Status 404 returned error can't find the container with id 21461e6cebbe69398713aeb0798997b88157bd0357c11eadc984855366f1d6ad
Nov 01 22:35:22.047592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.047560    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" event={"ID":"d8a585c4-897e-41d0-b122-19f8b56682e0","Type":"ContainerStarted","Data":"21461e6cebbe69398713aeb0798997b88157bd0357c11eadc984855366f1d6ad"}
Nov 01 22:35:22.049234 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.049205    5031 generic.go:356] "Generic (PLEG): container finished" podID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerID="813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d" exitCode=0
Nov 01 22:35:22.049382 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.049277    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerDied","Data":"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"}
Nov 01 22:35:22.049657 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.049639    5031 scope.go:122] "RemoveContainer" containerID="813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"
Nov 01 22:35:22.051006 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.050972    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:22.051137 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.051063    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:22.051137 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:22.051112    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:23.078980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.077136    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" event={"ID":"d8a585c4-897e-41d0-b122-19f8b56682e0","Type":"ContainerStarted","Data":"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"}
Nov 01 22:35:23.078980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.078185    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:23.081508 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.081477    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerStarted","Data":"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"}
Nov 01 22:35:23.195845 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.195728    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" podStartSLOduration=2.301849682 podStartE2EDuration="3.195707304s" totalImagesPullingTime="893.857622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:23.195597737 +0000 UTC m=+677.151637581" watchObservedRunningTime="2026-11-01 22:35:23.195707304 +0000 UTC m=+677.151747150"
Nov 01 22:35:23.312700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.312662    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:35:23.314018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.313973    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 22:35:23.314172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:23.314035    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 22:35:24.037134 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:24.037105    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 22:35:24.313259 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:24.313223    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 22:35:24.313710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:24.313284    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 22:35:24.900950 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:24.900913    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:35:25.313043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:25.313009    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 22:35:25.313230 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:25.313085    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 22:35:26.313026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:26.312908    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 22:35:26.313026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:26.312970    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 22:35:27.312889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:27.312853    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Nov 01 22:35:27.313069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:27.312920    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Nov 01 22:35:28.320657 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:28.320620    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:35:28.321731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:28.321709    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:35:30.096874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:30.096838    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 22:35:31.032110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.032074    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 22:35:31.037003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.036981    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.083810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.083775    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.084020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.083833    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.084020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.083924    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6sh\" (UniqueName: \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.084020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.083983    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.085479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.085460    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c768f3-serving-cert\""
Nov 01 22:35:31.133603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.133560    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 22:35:31.185475 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.185437    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.185773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.185504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.185773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.185531    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6sh\" (UniqueName: \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.185773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.185593    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.186211 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.186189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.186574 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:31.186524    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-serving-cert: secret "maas-api-e2e-worker-w2-c768f3-serving-cert" not found
Nov 01 22:35:31.186697 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:31.186635    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls podName:0a8d188a-d423-46b7-8370-efa2eb45ba5e nodeName:}" failed. No retries permitted until 2026-11-01 22:35:31.686612328 +0000 UTC m=+685.642652150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls") pod "maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" (UID: "0a8d188a-d423-46b7-8370-efa2eb45ba5e") : secret "maas-api-e2e-worker-w2-c768f3-serving-cert" not found
Nov 01 22:35:31.189325 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.189293    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.241583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.241528    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6sh\" (UniqueName: \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.693482 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.693443    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.696103 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.696083    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:31.949566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:31.949476    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:32.351722 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:32.351685    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 22:35:32.380620 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:35:32.380581    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8d188a_d423_46b7_8370_efa2eb45ba5e.slice/crio-9376ad47e8b0bb39b76c522a6fbda43af39df8e17e597bafa63064f0b40005cc WatchSource:0}: Error finding container 9376ad47e8b0bb39b76c522a6fbda43af39df8e17e597bafa63064f0b40005cc: Status 404 returned error can't find the container with id 9376ad47e8b0bb39b76c522a6fbda43af39df8e17e597bafa63064f0b40005cc
Nov 01 22:35:33.130332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:33.130295    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" event={"ID":"0a8d188a-d423-46b7-8370-efa2eb45ba5e","Type":"ContainerStarted","Data":"9376ad47e8b0bb39b76c522a6fbda43af39df8e17e597bafa63064f0b40005cc"}
Nov 01 22:35:34.137477 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:34.137438    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" event={"ID":"0a8d188a-d423-46b7-8370-efa2eb45ba5e","Type":"ContainerStarted","Data":"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"}
Nov 01 22:35:34.137916 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:34.137595    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:34.246788 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:34.246727    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" podStartSLOduration=3.301938529 podStartE2EDuration="4.246710525s" totalImagesPullingTime="944.771996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:34.24500637 +0000 UTC m=+688.201046274" watchObservedRunningTime="2026-11-01 22:35:34.246710525 +0000 UTC m=+688.202750369"
Nov 01 22:35:36.426909 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.426871    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 22:35:36.430909 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.430884    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.482027 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.481993    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-9dfe9f-serving-cert\""
Nov 01 22:35:36.528482 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.528436    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 22:35:36.546349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.546318    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7znnv\" (UniqueName: \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.546517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.546450    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.546517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.546490    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.546615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.546594    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.647518 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.647477    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7znnv\" (UniqueName: \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.647731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.647577    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.647731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.647607    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.647731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.647646    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.647895 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:36.647744    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-serving-cert: secret "maas-api-e2e-worker-w4-9dfe9f-serving-cert" not found
Nov 01 22:35:36.647895 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:36.647806    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls podName:03d934bf-46d8-4ec1-92d9-3ba4c7fb032b nodeName:}" failed. No retries permitted until 2026-11-01 22:35:37.147791744 +0000 UTC m=+691.103831579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls") pod "maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" (UID: "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b") : secret "maas-api-e2e-worker-w4-9dfe9f-serving-cert" not found
Nov 01 22:35:36.648310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.648283    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.650361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.650334    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:36.705208 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:36.705179    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7znnv\" (UniqueName: \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:37.153230 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:37.153191    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:37.156029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:37.155995    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:37.349827 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:37.349789    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:37.729373 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:37.729281    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 22:35:37.736076 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:35:37.736036    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d934bf_46d8_4ec1_92d9_3ba4c7fb032b.slice/crio-35702f0bd9d84433f77072e6b1d2328f25935302318e0b5d1f0d8c7c3c51bb90 WatchSource:0}: Error finding container 35702f0bd9d84433f77072e6b1d2328f25935302318e0b5d1f0d8c7c3c51bb90: Status 404 returned error can't find the container with id 35702f0bd9d84433f77072e6b1d2328f25935302318e0b5d1f0d8c7c3c51bb90
Nov 01 22:35:38.170673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:38.170636    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" event={"ID":"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b","Type":"ContainerStarted","Data":"35702f0bd9d84433f77072e6b1d2328f25935302318e0b5d1f0d8c7c3c51bb90"}
Nov 01 22:35:39.173524 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:39.173485    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" event={"ID":"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b","Type":"ContainerStarted","Data":"c0a6655a556f6afea9a183ac7c7ef47cd3c659ad02fb69be51c8b62fe2b17542"}
Nov 01 22:35:39.283269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:39.283215    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" podStartSLOduration=2.4535276599999998 podStartE2EDuration="3.283198957s" totalImagesPullingTime="829.671297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:39.282132885 +0000 UTC m=+693.238172746" watchObservedRunningTime="2026-11-01 22:35:39.283198957 +0000 UTC m=+693.239238801"
Nov 01 22:35:40.149938 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.149905    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 22:35:40.179416 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.179386    5031 generic.go:356] "Generic (PLEG): container finished" podID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerID="a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6" exitCode=0
Nov 01 22:35:40.179889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.179466    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerDied","Data":"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"}
Nov 01 22:35:40.179889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.179680    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:40.179996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.179983    5031 scope.go:122] "RemoveContainer" containerID="a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"
Nov 01 22:35:40.181189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.181161    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:40.181250 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.181233    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:40.181282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.181265    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:40.628421 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:35:40.628361    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:35:40.911627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:40.910657    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:35:41.204591 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:41.204473    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerStarted","Data":"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"}
Nov 01 22:35:41.823771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:41.823725    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:35:41.825276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:41.825246    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:35:41.825413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:41.825295    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 22:35:42.823116 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:42.823075    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:35:42.823631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:42.823131    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 22:35:43.823470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:43.823434    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:35:43.824042 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:43.823498    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 22:35:44.823491 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:44.823452    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:35:44.823908 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:44.823509    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 22:35:45.823081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:45.823035    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Nov 01 22:35:45.823289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:45.823116    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Nov 01 22:35:46.210483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:46.210453    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 22:35:46.828532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:46.828492    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:35:46.829053 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:46.829035    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:35:46.829935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:46.829916    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 22:35:49.248083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.248051    5031 generic.go:356] "Generic (PLEG): container finished" podID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerID="c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b" exitCode=0
Nov 01 22:35:49.248493 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.248120    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerDied","Data":"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"}
Nov 01 22:35:49.248558 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.248522    5031 scope.go:122] "RemoveContainer" containerID="c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"
Nov 01 22:35:49.249864 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.249832    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:49.249934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.249904    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:49.249969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:49.249937    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:35:50.279731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:50.279674    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerStarted","Data":"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"}
Nov 01 22:35:51.201557 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:51.201520    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:35:51.201741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:51.201636    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:35:51.201741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:51.201670    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 22:35:52.201302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:52.201266    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:35:52.201777 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:52.201316    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 22:35:53.201322 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:53.201283    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:35:53.201902 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:53.201338    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 22:35:54.200998 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:54.200960    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:35:54.201229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:54.201023    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 22:35:55.201200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:55.201162    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Nov 01 22:35:55.201674 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:55.201224    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Nov 01 22:35:56.205713 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:56.205682    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:35:56.206102 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:56.205971    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:35:56.206645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:35:56.206627    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 22:36:10.629792 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:36:10.629746    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:36:31.596121 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.596036    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 22:36:31.599999 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.599976    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.694515 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.694476    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"default-dockercfg-8s5xm\""
Nov 01 22:36:31.694717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.694622    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"simulator-w5-d6101b-kserve-self-signed-certs\""
Nov 01 22:36:31.696602 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.696575    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 22:36:31.706219 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706190    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.706219 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706229    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.706461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706271    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2664l\" (UniqueName: \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.706461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706414    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.706570 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706477    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.706648 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.706624    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.742335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.742297    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"kube-root-ca.crt\""
Nov 01 22:36:31.742335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.742328    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"openshift-service-ca.crt\""
Nov 01 22:36:31.808776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.808727    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.808964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.808797    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.808964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.808853    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2664l\" (UniqueName: \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.808964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.808907    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.808964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.808959    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.809131 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.809029    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.810274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.810241    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.810890 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.810822    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.811076 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.810941    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.813398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.813368    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.814356 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.814324    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.864160 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.864077    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2664l\" (UniqueName: \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\") pod \"simulator-w5-d6101b-kserve-65db569bfb-zchqf\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") " pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:31.912139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:31.912100    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:32.298365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:32.298328    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 22:36:32.303094 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:36:32.303053    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ebb0115_9339_49d9_8ae3_cf0cf0012000.slice/crio-0f5e7e75c28d81593b2e18683202a37feb24631f8837d904f015eecae4c5996f WatchSource:0}: Error finding container 0f5e7e75c28d81593b2e18683202a37feb24631f8837d904f015eecae4c5996f: Status 404 returned error can't find the container with id 0f5e7e75c28d81593b2e18683202a37feb24631f8837d904f015eecae4c5996f
Nov 01 22:36:32.502246 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:32.502042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" event={"ID":"1ebb0115-9339-49d9-8ae3-cf0cf0012000","Type":"ContainerStarted","Data":"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"}
Nov 01 22:36:32.502246 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:32.502116    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" event={"ID":"1ebb0115-9339-49d9-8ae3-cf0cf0012000","Type":"ContainerStarted","Data":"0f5e7e75c28d81593b2e18683202a37feb24631f8837d904f015eecae4c5996f"}
Nov 01 22:36:32.624896 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:32.624789    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podStartSLOduration=1.6247735140000001 podStartE2EDuration="1.624773514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:32.62265102 +0000 UTC m=+746.578690863" watchObservedRunningTime="2026-11-01 22:36:32.624773514 +0000 UTC m=+746.580813357"
Nov 01 22:36:32.912874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:32.912783    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:40.629347 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:36:40.629311    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:36:41.912549 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:41.912499    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:41.925795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:41.925767    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:42.550734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:42.549175    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:40074->10.132.0.35:15021: read: connection reset by peer" start-of-body=
Nov 01 22:36:42.550734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:42.549254    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": read tcp 10.132.0.2:40074->10.132.0.35:15021: read: connection reset by peer"
Nov 01 22:36:42.566300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:42.566273    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:36:43.561653 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.561616    5031 generic.go:356] "Generic (PLEG): container finished" podID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerID="521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0" exitCode=0
Nov 01 22:36:43.562080 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.561685    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerDied","Data":"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"}
Nov 01 22:36:43.562239 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.562224    5031 scope.go:122] "RemoveContainer" containerID="521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"
Nov 01 22:36:43.563474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.563441    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:36:43.563614 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.563528    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:36:43.563614 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:43.563597    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:36:44.585994 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:44.585953    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerStarted","Data":"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"}
Nov 01 22:36:44.865626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:44.857990    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:36:44.865626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:44.858126    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 22:36:44.865626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:44.858175    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 22:36:45.873567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:45.866006    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 22:36:45.873567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:45.866089    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 22:36:46.869050 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:46.869004    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 22:36:46.869728 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:46.869689    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 22:36:47.876566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:47.874013    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 22:36:47.876566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:47.874093    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 22:36:48.876565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:48.876020    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Nov 01 22:36:48.876565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:48.876100    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Nov 01 22:36:49.861223 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:49.861192    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:36:49.861591 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:49.861570    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:36:49.862339 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:36:49.862320    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 22:37:10.631497 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:37:10.631461    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:37:12.721141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.721104    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 22:37:12.725117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.725094    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.820385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.820350    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"openshift-service-ca.crt\""
Nov 01 22:37:12.820385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.820386    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"kube-root-ca.crt\""
Nov 01 22:37:12.820705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.820470    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"default-dockercfg-gptqw\""
Nov 01 22:37:12.820705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.820525    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"simulator-w6-a6dd2f-kserve-self-signed-certs\""
Nov 01 22:37:12.823819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.823786    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 22:37:12.893249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893212    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.893447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893270    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.893447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893344    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.893447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893371    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.893447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893398    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lft\" (UniqueName: \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.893653 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.893498    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994367    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994433    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994456    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994477    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994507    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lft\" (UniqueName: \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994581    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994907 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994913    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.994975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.994946    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.996908 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.996883    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:12.997204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:12.997187    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:13.049236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:13.049205    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lft\" (UniqueName: \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\") pod \"simulator-w6-a6dd2f-kserve-6779c6b844-cv898\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") " pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:13.337296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:13.337261    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:13.719090 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:13.718995    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 22:37:13.723648 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:37:13.723609    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2aec9d0_70f3_4823_8fa8_bce08fc89c8b.slice/crio-b5ed07b690d10e7e0ae78f65ba47d3328b3d4e22e91ef49ad88675e47bc7295f WatchSource:0}: Error finding container b5ed07b690d10e7e0ae78f65ba47d3328b3d4e22e91ef49ad88675e47bc7295f: Status 404 returned error can't find the container with id b5ed07b690d10e7e0ae78f65ba47d3328b3d4e22e91ef49ad88675e47bc7295f
Nov 01 22:37:13.741972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:13.741945    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" event={"ID":"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b","Type":"ContainerStarted","Data":"b5ed07b690d10e7e0ae78f65ba47d3328b3d4e22e91ef49ad88675e47bc7295f"}
Nov 01 22:37:14.748897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:14.748862    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" event={"ID":"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b","Type":"ContainerStarted","Data":"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"}
Nov 01 22:37:14.858867 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:14.858815    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podStartSLOduration=2.858802588 podStartE2EDuration="2.858802588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:14.856876776 +0000 UTC m=+788.812916620" watchObservedRunningTime="2026-11-01 22:37:14.858802588 +0000 UTC m=+788.814842432"
Nov 01 22:37:15.337652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:15.337618    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:23.337848 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:23.337785    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:23.350770 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:23.350742    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:23.805974 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:23.805944    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:37:29.770274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.770230    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 22:37:29.774275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.774251    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.869390 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.869346    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"default-dockercfg-nh9pl\""
Nov 01 22:37:29.869617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.869416    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"openshift-service-ca.crt\""
Nov 01 22:37:29.869617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.869435    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"premium-w3-e71d55-kserve-self-signed-certs\""
Nov 01 22:37:29.869617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.869461    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"kube-root-ca.crt\""
Nov 01 22:37:29.874203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.872394    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 22:37:29.874687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874652    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88xx\" (UniqueName: \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.874844 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.874844 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874759    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.874844 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874786    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.875023 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874856    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.875023 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.874906    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.975805 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975766    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88xx\" (UniqueName: \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975815    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975851    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975874    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975933    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.975970    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.976327    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.976412    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.976471 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.976434    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.978511 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.978486    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:29.978626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:29.978600    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:30.033638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.033570    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88xx\" (UniqueName: \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\") pod \"premium-w3-e71d55-kserve-5f949b8d74-g4wv8\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") " pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:30.087004 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.086962    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:30.467791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.467748    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 22:37:30.473413 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:37:30.473384    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0e01ef8_e44c_41c2_8e4c_76b753c8c410.slice/crio-71d8b9fe2bf415b678c77b43952cb4900f5c5bf3a24323d9769140f1bacaaf15 WatchSource:0}: Error finding container 71d8b9fe2bf415b678c77b43952cb4900f5c5bf3a24323d9769140f1bacaaf15: Status 404 returned error can't find the container with id 71d8b9fe2bf415b678c77b43952cb4900f5c5bf3a24323d9769140f1bacaaf15
Nov 01 22:37:30.834632 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.834595    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" event={"ID":"a0e01ef8-e44c-41c2-8e4c-76b753c8c410","Type":"ContainerStarted","Data":"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"}
Nov 01 22:37:30.834632 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.834639    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" event={"ID":"a0e01ef8-e44c-41c2-8e4c-76b753c8c410","Type":"ContainerStarted","Data":"71d8b9fe2bf415b678c77b43952cb4900f5c5bf3a24323d9769140f1bacaaf15"}
Nov 01 22:37:30.947500 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:30.947446    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podStartSLOduration=1.9474316329999999 podStartE2EDuration="1.947431633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:30.947061989 +0000 UTC m=+804.903101832" watchObservedRunningTime="2026-11-01 22:37:30.947431633 +0000 UTC m=+804.903471476"
Nov 01 22:37:31.087411 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:31.087325    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:40.087723 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:40.087685    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:40.100489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:40.100460    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:37:40.626016 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:37:40.625969    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:37:40.899934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:37:40.899848    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:38:10.626970 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:38:10.626928    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:38:19.571956 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.571921    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 22:38:19.575884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.575856    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.624717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.624682    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"kube-root-ca.crt\""
Nov 01 22:38:19.624717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.624700    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"openshift-service-ca.crt\""
Nov 01 22:38:19.670517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.670486    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"default-dockercfg-wcfqv\""
Nov 01 22:38:19.670517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.670486    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"premium-w4-9dfe9f-kserve-self-signed-certs\""
Nov 01 22:38:19.673977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.673944    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 22:38:19.724844 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.724791    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.725058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.724978    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.725058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.725047    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.725172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.725150    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.727195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.725319    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7sr\" (UniqueName: \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.727195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.725370    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.826970 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.826874    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7sr\" (UniqueName: \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.826970 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.826922    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.826970 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.826954    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827017    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827037    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827079    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827455    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.827580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.827515    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.829641 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.829613    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.829865 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.829848    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.881018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.880979    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7sr\" (UniqueName: \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\") pod \"premium-w4-9dfe9f-kserve-f5fcffb95-n99gm\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") " pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:19.887706 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:19.887670    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:20.292513 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:20.292485    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 22:38:20.297311 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:38:20.297275    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d4580fb_35f4_4d7f_afcc_a00232349ce8.slice/crio-361d06de2f5ac55ce55e2ce0095e67ae9610909b365625ed3d5b9e63f68cd2f5 WatchSource:0}: Error finding container 361d06de2f5ac55ce55e2ce0095e67ae9610909b365625ed3d5b9e63f68cd2f5: Status 404 returned error can't find the container with id 361d06de2f5ac55ce55e2ce0095e67ae9610909b365625ed3d5b9e63f68cd2f5
Nov 01 22:38:21.112733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:21.112699    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" event={"ID":"7d4580fb-35f4-4d7f-afcc-a00232349ce8","Type":"ContainerStarted","Data":"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"}
Nov 01 22:38:21.112733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:21.112743    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" event={"ID":"7d4580fb-35f4-4d7f-afcc-a00232349ce8","Type":"ContainerStarted","Data":"361d06de2f5ac55ce55e2ce0095e67ae9610909b365625ed3d5b9e63f68cd2f5"}
Nov 01 22:38:21.224572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:21.224499    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podStartSLOduration=2.224483488 podStartE2EDuration="2.224483488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:21.222478408 +0000 UTC m=+855.178518253" watchObservedRunningTime="2026-11-01 22:38:21.224483488 +0000 UTC m=+855.180523331"
Nov 01 22:38:21.888601 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:21.888524    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:29.888530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:29.888486    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:29.902221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:29.902193    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:30.171842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:30.171764    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:38:40.627887 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:38:40.627843    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:38:57.176926 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.176892    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:38:57.180972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.180950    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.239067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.239026    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9dfe9f-openshift-default-dockercfg-9r4dj\""
Nov 01 22:38:57.272100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272065    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272192    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272242    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272269    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272289    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272371    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272515 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272428    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh42g\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272515 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272493    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.272673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.272602    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.296022 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.295987    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:38:57.373956 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.373920    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374005    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374044    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374065    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374085    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374117    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh42g\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374181    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374231    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374579    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374694    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.374937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374881    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.375015 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.374869    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.375122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.375096    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.376571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.376550    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.377161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.377140    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.428170 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.428091    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.428308 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.428224    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh42g\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.494689 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.494652    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:57.925633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.925595    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:38:57.932456 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:38:57.932421    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53be7b1c_747d_4375_b79e_1550be77f7d5.slice/crio-036468aa808ed6b271ee61f41c055d9b60446e8187be8e8890799fb062e3a157 WatchSource:0}: Error finding container 036468aa808ed6b271ee61f41c055d9b60446e8187be8e8890799fb062e3a157: Status 404 returned error can't find the container with id 036468aa808ed6b271ee61f41c055d9b60446e8187be8e8890799fb062e3a157
Nov 01 22:38:57.934890 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.934857    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:38:57.934968 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.934925    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:38:57.934968 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:57.934957    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:38:58.305014 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.304976    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" event={"ID":"53be7b1c-747d-4375-b79e-1550be77f7d5","Type":"ContainerStarted","Data":"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"}
Nov 01 22:38:58.305404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.305031    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" event={"ID":"53be7b1c-747d-4375-b79e-1550be77f7d5","Type":"ContainerStarted","Data":"036468aa808ed6b271ee61f41c055d9b60446e8187be8e8890799fb062e3a157"}
Nov 01 22:38:58.425754 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.425699    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podStartSLOduration=1.425685283 podStartE2EDuration="1.425685283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:58.424153553 +0000 UTC m=+892.380193396" watchObservedRunningTime="2026-11-01 22:38:58.425685283 +0000 UTC m=+892.381725126"
Nov 01 22:38:58.495737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.495703    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:38:58.497100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.497072    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:38:58.497209 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:58.497124    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:38:59.495470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:59.495437    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:38:59.495928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:38:59.495497    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:00.495737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:00.495694    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:39:00.496260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:00.495772    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:01.494990 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:01.494955    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:39:01.495201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:01.495018    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:02.495914 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:02.495877    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:39:02.496432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:02.495941    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:03.495158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:03.495123    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:39:03.495435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:03.495188    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:04.496265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:04.495797    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Nov 01 22:39:04.496265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:04.495862    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Nov 01 22:39:05.504329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:05.504277    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:39:05.504817 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:05.504779    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:39:05.505767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:05.505744    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:39:10.007397 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.007356    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:39:10.011387 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.011362    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.072395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.072363    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c54de0-openshift-default-dockercfg-k2kn2\""
Nov 01 22:39:10.115635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.115587    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151142    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151199    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151227    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151260    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151300    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151464    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42w7\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151565    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151705    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.152079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.151739    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.254011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.253945    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.254292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.254272    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256594 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.255147    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256639    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256706    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256856 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256751    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256856 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256809    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42w7\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.256973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256880    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.256988    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.257021    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.257381    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.257687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.257869    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.257953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.257930    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.259574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.259510    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.259777 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.259758    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.307459 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.307430    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.310246 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.310220    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42w7\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.324912 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.324883    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:10.626660 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:39:10.626619    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:39:10.751467 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.751432    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:39:10.756660 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:39:10.756618    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcabe3e_1f3b_4471_904d_fd934a18a547.slice/crio-9e14e025ea809e61a728ff8f22550848e3a52b55660568dbcf981e82ffa39860 WatchSource:0}: Error finding container 9e14e025ea809e61a728ff8f22550848e3a52b55660568dbcf981e82ffa39860: Status 404 returned error can't find the container with id 9e14e025ea809e61a728ff8f22550848e3a52b55660568dbcf981e82ffa39860
Nov 01 22:39:10.759083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.759049    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:10.759166 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.759116    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:10.759166 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:10.759148    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:11.385702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:11.385661    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" event={"ID":"6dcabe3e-1f3b-4471-904d-fd934a18a547","Type":"ContainerStarted","Data":"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"}
Nov 01 22:39:11.386215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:11.385725    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" event={"ID":"6dcabe3e-1f3b-4471-904d-fd934a18a547","Type":"ContainerStarted","Data":"9e14e025ea809e61a728ff8f22550848e3a52b55660568dbcf981e82ffa39860"}
Nov 01 22:39:11.496707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:11.496641    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podStartSLOduration=2.496627402 podStartE2EDuration="2.496627402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:11.494946405 +0000 UTC m=+905.450986249" watchObservedRunningTime="2026-11-01 22:39:11.496627402 +0000 UTC m=+905.452667246"
Nov 01 22:39:12.325346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:12.325312    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:12.327136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:12.327109    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 22:39:12.327267 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:12.327157    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 22:39:13.325552 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:13.325513    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 22:39:13.325938 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:13.325592    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 22:39:14.325381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:14.325340    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 22:39:14.325589 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:14.325408    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 22:39:15.325841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:15.325801    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 22:39:15.326347 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:15.325866    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 22:39:16.326053 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:16.326015    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Nov 01 22:39:16.326463 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:16.326070    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Nov 01 22:39:17.601759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:17.601720    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:17.602361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:17.602338    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:17.603428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:17.603408    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:39:40.625946 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:39:40.625899    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:39:54.958606 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:54.958568    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 22:39:54.962829 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:54.962797    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.013283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.013249    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e71d55-openshift-default-dockercfg-696c5\""
Nov 01 22:39:55.063254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.063219    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 22:39:55.067881 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.067842    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068049 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.067952    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068049 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068012    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflzx\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068143    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068179    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068292 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068206    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068288    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068351    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.068463 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.068426    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.170713 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.170679    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.170885 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.170720    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.170885 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.170833    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflzx\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.170885 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.170878    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171072 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171052    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171098    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171199 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171178    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171250 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171183    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171267    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171324 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171291    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171349    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171412    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.171897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.171878    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.173472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.173454    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.173833 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.173814    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.226295 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.226214    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.226457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.226322    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflzx\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.279099 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.279068    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:55.725958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.725921    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 22:39:55.733911 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:39:55.733878    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c41ea7_baab_49f6_801d_6043182ee02e.slice/crio-113cf42ddd07354d36716e93a935b9afdb009bb1fbf4f148bf275062adc52e08 WatchSource:0}: Error finding container 113cf42ddd07354d36716e93a935b9afdb009bb1fbf4f148bf275062adc52e08: Status 404 returned error can't find the container with id 113cf42ddd07354d36716e93a935b9afdb009bb1fbf4f148bf275062adc52e08
Nov 01 22:39:55.737220 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.737176    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:55.737353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.737257    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:55.737353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:55.737290    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:39:56.713564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:56.712979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" event={"ID":"e4c41ea7-baab-49f6-801d-6043182ee02e","Type":"ContainerStarted","Data":"ee9a3c19dc87d505ac6903b96ff49eff7dcf6c12027eaa4988195aca5035ab31"}
Nov 01 22:39:56.713564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:56.713106    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" event={"ID":"e4c41ea7-baab-49f6-801d-6043182ee02e","Type":"ContainerStarted","Data":"113cf42ddd07354d36716e93a935b9afdb009bb1fbf4f148bf275062adc52e08"}
Nov 01 22:39:56.944862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:56.944810    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podStartSLOduration=2.94479613 podStartE2EDuration="2.94479613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:56.908872593 +0000 UTC m=+950.864912439" watchObservedRunningTime="2026-11-01 22:39:56.94479613 +0000 UTC m=+950.900835973"
Nov 01 22:39:57.280969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:57.280934    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:39:57.297567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:57.290718    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:39:57.297567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:57.290822    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:39:58.283055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:58.283010    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:39:58.283727 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:58.283691    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:39:59.283114 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:59.283073    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:39:59.283751 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:39:59.283717    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:40:00.280050 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:00.280009    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:40:00.280389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:00.280067    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:40:01.279666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:01.279608    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:40:01.280082 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:01.279706    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:40:02.280586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:02.280513    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Nov 01 22:40:02.280995 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:02.280603    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Nov 01 22:40:03.285028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:03.285000    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:40:03.285444 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:03.285271    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:40:03.286151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:03.286132    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 22:40:10.625892 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:40:10.625858    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:40:24.859327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.859290    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:40:24.863660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.863636    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949283    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949339    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949424    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949485    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949504    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949596    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949687    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.949819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.949720    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djt79\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:24.969740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:24.969701    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:40:25.051603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051556    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.051814 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051624    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.051814 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051733    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.051814 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051769    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djt79\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.051982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051898    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.051982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052089 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.051983    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052089 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.052022    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052089 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.052053    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052089 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.052070    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.052192    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.052998 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.052968    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.053175 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.053087    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.053302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.053277    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.054918 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.054889    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.054918 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.054910    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.106083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.106046    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.106261 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.106192    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djt79\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.178875 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.178784    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:25.609980 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.609945    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:40:25.614255 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:40:25.614226    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d484274_310f_4149_be9d_6a62c1f8ae93.slice/crio-2aaec39896a96b85da7026f0e612e9f22d21958d34c63bebb98d5fb45c2287e2 WatchSource:0}: Error finding container 2aaec39896a96b85da7026f0e612e9f22d21958d34c63bebb98d5fb45c2287e2: Status 404 returned error can't find the container with id 2aaec39896a96b85da7026f0e612e9f22d21958d34c63bebb98d5fb45c2287e2
Nov 01 22:40:25.616816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.616781    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:25.616884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.616856    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:25.616927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.616885    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:25.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.880416    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" event={"ID":"7d484274-310f-4149-be9d-6a62c1f8ae93","Type":"ContainerStarted","Data":"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"}
Nov 01 22:40:25.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.880480    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" event={"ID":"7d484274-310f-4149-be9d-6a62c1f8ae93","Type":"ContainerStarted","Data":"2aaec39896a96b85da7026f0e612e9f22d21958d34c63bebb98d5fb45c2287e2"}
Nov 01 22:40:25.995527 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:25.995470    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podStartSLOduration=1.995455515 podStartE2EDuration="1.995455515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:25.993429939 +0000 UTC m=+979.949469807" watchObservedRunningTime="2026-11-01 22:40:25.995455515 +0000 UTC m=+979.951495349"
Nov 01 22:40:26.181451 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:26.181355    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:26.181451 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:26.181420    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:26.183998 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:26.183966    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:27.180125 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:27.180082    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:27.180666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:27.180147    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:28.179152 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:28.179111    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:28.179360 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:28.179185    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:29.179342 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:29.179303    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:29.179826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:29.179365    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:30.182327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:30.182285    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:30.182840 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:30.182350    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:31.180079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:31.180038    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 01 22:40:31.180422 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:31.180112    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 01 22:40:32.388157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:32.388121    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:32.388680 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:32.388479    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:32.389333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:32.389315    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:40:39.867483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.867438    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:40:39.872695 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.872668    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960195    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960228    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960256    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960388    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn62\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960445    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960498    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960593    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.960715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.960666    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:39.977817 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:39.977779    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:40:40.062261 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062221    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062478    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062579    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062671 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062660    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062747    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062772    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062808    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.062899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.062864    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn62\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.063095 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.063070    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.063226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.063203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.063296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.063236    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.063343 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.063323    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.063606 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.063591    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.065172 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.065146    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.065377 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.065360    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.117598 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.117488    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.117733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.117633    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn62\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.193702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.193664    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:40.626878 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:40:40.626842    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:40:40.626878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.626873    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:40:40.632166 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:40:40.632140    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6212b9_7ec6_4561_9017_5286c3a0efb4.slice/crio-884139e916e023e2cd62f938a5ddba59658ddac86833f741a7ef9a89fac79a2d WatchSource:0}: Error finding container 884139e916e023e2cd62f938a5ddba59658ddac86833f741a7ef9a89fac79a2d: Status 404 returned error can't find the container with id 884139e916e023e2cd62f938a5ddba59658ddac86833f741a7ef9a89fac79a2d
Nov 01 22:40:40.634478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.634446    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:40.634591 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.634514    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:40.634591 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.634565    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:40:40.973283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.973183    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" event={"ID":"bf6212b9-7ec6-4561-9017-5286c3a0efb4","Type":"ContainerStarted","Data":"c45edfdbc8ebb8583362c7439a9a03d021993cd3f3a0716a5876af90cd3b58fa"}
Nov 01 22:40:40.973283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:40.973257    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" event={"ID":"bf6212b9-7ec6-4561-9017-5286c3a0efb4","Type":"ContainerStarted","Data":"884139e916e023e2cd62f938a5ddba59658ddac86833f741a7ef9a89fac79a2d"}
Nov 01 22:40:41.086195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:41.086140    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podStartSLOduration=2.0861264840000002 podStartE2EDuration="2.086126484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:41.084480074 +0000 UTC m=+995.040519917" watchObservedRunningTime="2026-11-01 22:40:41.086126484 +0000 UTC m=+995.042166328"
Nov 01 22:40:41.194322 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:41.194275    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:41.196718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:41.196682    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:41.196898 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:41.196763    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:42.194015 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:42.193981    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:42.194409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:42.194034    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:43.194300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:43.194263    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:43.194701 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:43.194343    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:44.194504 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:44.194469    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:44.194932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:44.194532    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:45.194501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:45.194457    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:45.194759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:45.194531    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:46.194293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:46.194258    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:46.194473 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:46.194327    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:47.194466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:47.194429    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Nov 01 22:40:47.194922 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:47.194496    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Nov 01 22:40:48.200103 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:48.200071    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:48.200618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:48.200455    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:48.201099 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:48.201077    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:40:53.991303 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:53.991264    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 22:40:53.995554 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:53.995522    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049476 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049435    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ncm\" (UniqueName: \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049671    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049793    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049814    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.049904 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.049892    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.090834 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.090800    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"distinct-w4-9dfe9f-kserve-self-signed-certs\""
Nov 01 22:40:54.094720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.094685    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 22:40:54.151151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151112    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151213    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ncm\" (UniqueName: \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151298    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151350    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151454    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151477    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151623    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.151963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.151933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.152147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.152126    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.154003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.153972    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.154300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.154281    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.205508 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.205474    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ncm\" (UniqueName: \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\") pod \"distinct-w4-9dfe9f-kserve-7889c58778-s6dpf\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.308380 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.308340    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:40:54.726957 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:54.726878    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 22:40:54.732376 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:40:54.732342    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7743905f_5888_4188_8ace_fd72d94cdec3.slice/crio-8a93743d9d15428f3f19abc7d3be0ce3edd90cdf6f6105e75010a2cf5a2c3acd WatchSource:0}: Error finding container 8a93743d9d15428f3f19abc7d3be0ce3edd90cdf6f6105e75010a2cf5a2c3acd: Status 404 returned error can't find the container with id 8a93743d9d15428f3f19abc7d3be0ce3edd90cdf6f6105e75010a2cf5a2c3acd
Nov 01 22:40:55.059034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:55.059002    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" event={"ID":"7743905f-5888-4188-8ace-fd72d94cdec3","Type":"ContainerStarted","Data":"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"}
Nov 01 22:40:55.059034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:55.059042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" event={"ID":"7743905f-5888-4188-8ace-fd72d94cdec3","Type":"ContainerStarted","Data":"8a93743d9d15428f3f19abc7d3be0ce3edd90cdf6f6105e75010a2cf5a2c3acd"}
Nov 01 22:40:55.171767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:55.171714    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podStartSLOduration=2.17170099 podStartE2EDuration="2.17170099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:55.171188557 +0000 UTC m=+1009.127228403" watchObservedRunningTime="2026-11-01 22:40:55.17170099 +0000 UTC m=+1009.127740833"
Nov 01 22:40:55.309206 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:40:55.308994    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:41:04.309445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:04.309402    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:41:04.323215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:04.323181    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:41:05.132257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:05.132225    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:41:10.625856 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:41:10.625815    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:41:28.380459 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.380427    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 22:41:28.384568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.384514    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.483270 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.483223    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"openshift-service-ca.crt\""
Nov 01 22:41:28.483270 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.483234    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"kube-root-ca.crt\""
Nov 01 22:41:28.483499 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.483277    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"default-dockercfg-72fgz\""
Nov 01 22:41:28.483499 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.483331    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b8fd2\"/\"distinct-w1-5b8fd2-kserve-self-signed-certs\""
Nov 01 22:41:28.485746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.485726    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 22:41:28.514400 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514364    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.514618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514418    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vsj\" (UniqueName: \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.514618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514517    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.514618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514606    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.514749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.514749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.514735    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616009    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616092    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616195    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616231    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616452 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616274    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vsj\" (UniqueName: \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616452 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616405    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616607 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616525    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.616707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.616687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.618719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.618698    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.618921 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.618904    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.672718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.672633    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vsj\" (UniqueName: \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\") pod \"distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") " pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:28.696361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:28.696318    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:29.076361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:29.076324    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 22:41:29.081686 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:41:29.081654    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd48b3c0a_9e20_40f5_b441_4109ded549c9.slice/crio-61479aed73057a2a2bb518d0f5477f92e66d0e30df60e839969a45313a8e3c4c WatchSource:0}: Error finding container 61479aed73057a2a2bb518d0f5477f92e66d0e30df60e839969a45313a8e3c4c: Status 404 returned error can't find the container with id 61479aed73057a2a2bb518d0f5477f92e66d0e30df60e839969a45313a8e3c4c
Nov 01 22:41:29.274123 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:29.274083    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" event={"ID":"d48b3c0a-9e20-40f5-b441-4109ded549c9","Type":"ContainerStarted","Data":"a618ddc0605cc07e988003575d4e133512d6d6a49d8f1c150ab464f9fade084b"}
Nov 01 22:41:29.274123 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:29.274126    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" event={"ID":"d48b3c0a-9e20-40f5-b441-4109ded549c9","Type":"ContainerStarted","Data":"61479aed73057a2a2bb518d0f5477f92e66d0e30df60e839969a45313a8e3c4c"}
Nov 01 22:41:29.386561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:29.386447    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podStartSLOduration=1.386431827 podStartE2EDuration="1.386431827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:29.383617156 +0000 UTC m=+1043.339656992" watchObservedRunningTime="2026-11-01 22:41:29.386431827 +0000 UTC m=+1043.342471671"
Nov 01 22:41:29.696718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:29.696630    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:38.696807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:38.696777    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:38.709157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:38.709133    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:39.342074 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:39.342044    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:41:40.626528 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:41:40.626492    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:41:43.530501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.530465    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:41:43.534804 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.534773    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.598642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.598601    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:41:43.599631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599596    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.599788 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599665    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.599788 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599768    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.599915 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599823    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.600016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599930    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.600016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.599985    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb28h\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.600136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.600030    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.600190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.600151    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.600273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.600203    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701368 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701333    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701381    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701435    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701476    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701500    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701553    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701608    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701944 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701638    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb28h\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.701944 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.701684    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.702106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.702079    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.702174 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.702105    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.702254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.702191    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.702254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.702231    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.702556 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.702513    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.704036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.704017    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.704281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.704262    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.754555 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.754517    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.754762 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.754741    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb28h\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:43.853917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:43.853878    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:44.280650 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:44.280512    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:41:44.323591 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:41:44.323559    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaef116d_b8ec_4816_96a0_c69f81eb2f62.slice/crio-9f52eb5b4ce838d44bfbdf510e070b0ea5a63773593587e9f3daa33069eb0c45 WatchSource:0}: Error finding container 9f52eb5b4ce838d44bfbdf510e070b0ea5a63773593587e9f3daa33069eb0c45: Status 404 returned error can't find the container with id 9f52eb5b4ce838d44bfbdf510e070b0ea5a63773593587e9f3daa33069eb0c45
Nov 01 22:41:44.326267 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:44.326232    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:44.326336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:44.326306    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:44.326374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:44.326340    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:44.360020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:44.359988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" event={"ID":"caef116d-b8ec-4816-96a0-c69f81eb2f62","Type":"ContainerStarted","Data":"9f52eb5b4ce838d44bfbdf510e070b0ea5a63773593587e9f3daa33069eb0c45"}
Nov 01 22:41:45.382551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:45.382484    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" event={"ID":"caef116d-b8ec-4816-96a0-c69f81eb2f62","Type":"ContainerStarted","Data":"aeb12efbfff8b3cadc3cf1a2e5b145411b7c211efbeafe98b2ff5136fffdddd7"}
Nov 01 22:41:45.588248 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:45.588187    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podStartSLOduration=2.588173109 podStartE2EDuration="2.588173109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:45.54099173 +0000 UTC m=+1059.497031575" watchObservedRunningTime="2026-11-01 22:41:45.588173109 +0000 UTC m=+1059.544213031"
Nov 01 22:41:45.862736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:45.862700    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:45.872566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:45.868677    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:45.872566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:45.868800    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:46.874567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:46.868813    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:46.874567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:46.868899    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:47.861566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:47.856030    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:47.861566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:47.856107    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:48.863058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:48.863012    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:48.863763 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:48.863723    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:49.871567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:49.866957    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:49.871567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:49.867018    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:50.854796 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:50.854743    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 01 22:41:50.855011 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:50.854828    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 01 22:41:51.859251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:51.859220    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:51.859689 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:51.859644    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:51.860600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:51.860573    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:41:55.117954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.117917    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:41:55.121501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.121482    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275412 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275352    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275524    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275581    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275700    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275777 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275750    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6pk\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275806    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275835    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275892 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275867    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.275975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.275961    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.368246 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.368161    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:41:55.376912 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.376878    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377193 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377169    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377390    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377503    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377682    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377439    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377835 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377531    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.377939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.377920    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.378056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.378038    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6pk\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.378207 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.378189    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.378724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.378691    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.378850 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.378622    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.378992 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.378975    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.379643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.379618    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.380803 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.380776    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.381690 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.381669    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.434676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.434643    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.434820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.434724    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6pk\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.446919 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.446890    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:55.873763 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.873726    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:41:55.878711 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:41:55.878685    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90369e1_46be_40a6_825a_c30caf3481a6.slice/crio-1453b89d7f4488ab71c148d5fcad8ae032a9eee97480a8fca7365c669d33487f WatchSource:0}: Error finding container 1453b89d7f4488ab71c148d5fcad8ae032a9eee97480a8fca7365c669d33487f: Status 404 returned error can't find the container with id 1453b89d7f4488ab71c148d5fcad8ae032a9eee97480a8fca7365c669d33487f
Nov 01 22:41:55.881268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.881234    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:55.881325 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.881305    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:55.881360 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:55.881335    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:41:56.463031 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:56.462900    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" event={"ID":"c90369e1-46be-40a6-825a-c30caf3481a6","Type":"ContainerStarted","Data":"3d729305073c40904a5a9cb87c5b7d53fa815eb95a3c5316362c8b77f80fd849"}
Nov 01 22:41:56.463793 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:56.463768    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" event={"ID":"c90369e1-46be-40a6-825a-c30caf3481a6","Type":"ContainerStarted","Data":"1453b89d7f4488ab71c148d5fcad8ae032a9eee97480a8fca7365c669d33487f"}
Nov 01 22:41:56.622178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:56.622123    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podStartSLOduration=1.6221063039999999 podStartE2EDuration="1.622106304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:56.620143593 +0000 UTC m=+1070.576183436" watchObservedRunningTime="2026-11-01 22:41:56.622106304 +0000 UTC m=+1070.578146215"
Nov 01 22:41:57.448088 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:57.448050    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:41:57.450049 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:57.450019    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:41:57.450191 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:57.450081    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:41:58.447815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:58.447780    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:41:58.448262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:58.447835    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:41:59.447965 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:59.447923    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:41:59.448396 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:41:59.447982    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:42:00.448285 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:00.448247    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:42:00.448718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:00.448305    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:42:01.447938 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:01.447898    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:42:01.448128 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:01.447954    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:42:02.447509 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:02.447470    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 01 22:42:02.447957 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:02.447554    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 01 22:42:03.451892 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:03.451857    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:42:03.452369 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:03.452155    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:42:03.452879 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:03.452865    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:42:10.625813 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:42:10.625774    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:42:33.491097 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.491011    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 22:42:33.495331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.495307    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.591617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.591574    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"distinct-2-w6-a6dd2f-kserve-self-signed-certs\""
Nov 01 22:42:33.595142 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595109    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595175    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595336    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595404 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595362    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxjw\" (UniqueName: \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595594 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595494    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.595860 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.595832    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 22:42:33.697529 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.697486    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.697763 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.697680    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.697841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.697790    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.697841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.697834    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxjw\" (UniqueName: \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.697947 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.697905    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.698314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.698041    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.698579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.698524    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.699090 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.699047    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.699363 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.699332    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.702585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.702505    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.703783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.703759    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.754040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.753969    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxjw\" (UniqueName: \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\") pod \"distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:33.809266 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:33.809232    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:34.224104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:34.224067    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 22:42:34.230593 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:42:34.230562    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8abdde21_b92d_44bf_8454_b346bf136a24.slice/crio-96d2d18896db1d6d651bfe0a039b2286f57d69d4b1fc601b685bcffffc89e580 WatchSource:0}: Error finding container 96d2d18896db1d6d651bfe0a039b2286f57d69d4b1fc601b685bcffffc89e580: Status 404 returned error can't find the container with id 96d2d18896db1d6d651bfe0a039b2286f57d69d4b1fc601b685bcffffc89e580
Nov 01 22:42:34.716861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:34.716819    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" event={"ID":"8abdde21-b92d-44bf-8454-b346bf136a24","Type":"ContainerStarted","Data":"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"}
Nov 01 22:42:34.717271 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:34.716887    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" event={"ID":"8abdde21-b92d-44bf-8454-b346bf136a24","Type":"ContainerStarted","Data":"96d2d18896db1d6d651bfe0a039b2286f57d69d4b1fc601b685bcffffc89e580"}
Nov 01 22:42:34.809971 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:34.809935    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:34.829220 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:34.829156    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podStartSLOduration=1.829139127 podStartE2EDuration="1.829139127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:34.826367225 +0000 UTC m=+1108.782407081" watchObservedRunningTime="2026-11-01 22:42:34.829139127 +0000 UTC m=+1108.785178970"
Nov 01 22:42:40.625887 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:42:40.625849    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:42:43.809741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:43.809702    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:43.822235 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:43.822211    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:42:44.786199 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:42:44.786171    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:43:10.579318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.579281    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 22:43:10.583683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.583658    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.627457 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:43:10.627419    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:43:10.680280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680243    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680491 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680337    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dws\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680491 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680395    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680491 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680416    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680581    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680704    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680857 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680763    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680857 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680826    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.680983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.680964    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.686332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.686304    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 22:43:10.782789 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782751    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.782789 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782804    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dws\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782860    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782880    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782915    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.782975    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783027    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.783059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783053    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783144    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783235    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783937    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.783962    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.784124    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.784656 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.784376    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.786687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.786182    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.786687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.786303    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.837919 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.837848    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.838062 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.837930    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dws\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:10.897682 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:10.897651    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:11.333499 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.333464    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 22:43:11.339440 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:43:11.339406    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766d5209_2eac_4d97_81bc_4a5b9f9d8608.slice/crio-b435ef3a1dd449548516e2e79f8446fbea47307fa0f8f3c149467607f3142a50 WatchSource:0}: Error finding container b435ef3a1dd449548516e2e79f8446fbea47307fa0f8f3c149467607f3142a50: Status 404 returned error can't find the container with id b435ef3a1dd449548516e2e79f8446fbea47307fa0f8f3c149467607f3142a50
Nov 01 22:43:11.342115 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.342078    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:11.342226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.342168    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:11.342226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.342215    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:11.945354 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.945316    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" event={"ID":"766d5209-2eac-4d97-81bc-4a5b9f9d8608","Type":"ContainerStarted","Data":"e346eaca0820214d049bd081c37c6828a9c569b0a0001e69a7f8934f57e25238"}
Nov 01 22:43:11.945784 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:11.945375    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" event={"ID":"766d5209-2eac-4d97-81bc-4a5b9f9d8608","Type":"ContainerStarted","Data":"b435ef3a1dd449548516e2e79f8446fbea47307fa0f8f3c149467607f3142a50"}
Nov 01 22:43:12.063240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:12.063179    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podStartSLOduration=2.063163511 podStartE2EDuration="2.063163511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:12.059952737 +0000 UTC m=+1146.015992580" watchObservedRunningTime="2026-11-01 22:43:12.063163511 +0000 UTC m=+1146.019203355"
Nov 01 22:43:12.898838 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:12.898800    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:12.900887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:12.900858    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:12.901065 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:12.900911    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:13.898679 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:13.898645    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:13.899093 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:13.898699    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:14.898493 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:14.898454    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:14.898866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:14.898533    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:15.898652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:15.898617    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:15.898849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:15.898696    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:16.898531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:16.898495    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:16.898921 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:16.898580    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:17.898071 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:17.898028    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:43:17.898284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:17.898110    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 01 22:43:19.048460 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:19.048423    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:19.049451 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:19.049423    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:19.050475 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:19.050452    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 22:43:28.223190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.223155    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 22:43:28.227217 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.227195    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246180    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246244    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczwq\" (UniqueName: \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246280    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246372    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246469    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.246622 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.246604    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.326382 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.326352    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-e71d55\"/\"distinct-2-w3-e71d55-kserve-self-signed-certs\""
Nov 01 22:43:28.329222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.329195    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 22:43:28.347414 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347373    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347506    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347556    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczwq\" (UniqueName: \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347732    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347743    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347865 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347773    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.347865 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.347811    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.348123 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.348100    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.348430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.348187    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.350069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.350047    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.350171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.350139    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.403530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.403485    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczwq\" (UniqueName: \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\") pod \"distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") " pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.540460 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.540371    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:28.979203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:28.979168    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 22:43:28.984835 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:43:28.984807    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f89fda_9319_49db_90c8_a91abd4318d1.slice/crio-69a1945c9ed9e62278d06c718d92f9293d980327c2afff0de773afede690bdde WatchSource:0}: Error finding container 69a1945c9ed9e62278d06c718d92f9293d980327c2afff0de773afede690bdde: Status 404 returned error can't find the container with id 69a1945c9ed9e62278d06c718d92f9293d980327c2afff0de773afede690bdde
Nov 01 22:43:29.091183 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:29.091136    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" event={"ID":"10f89fda-9319-49db-90c8-a91abd4318d1","Type":"ContainerStarted","Data":"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"}
Nov 01 22:43:29.091408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:29.091198    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" event={"ID":"10f89fda-9319-49db-90c8-a91abd4318d1","Type":"ContainerStarted","Data":"69a1945c9ed9e62278d06c718d92f9293d980327c2afff0de773afede690bdde"}
Nov 01 22:43:29.204429 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:29.204373    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podStartSLOduration=1.204356216 podStartE2EDuration="1.204356216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:29.203015863 +0000 UTC m=+1163.159055707" watchObservedRunningTime="2026-11-01 22:43:29.204356216 +0000 UTC m=+1163.160396253"
Nov 01 22:43:29.541057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:29.541019    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:38.541086 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:38.541052    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:38.553839 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:38.553810    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:39.186798 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:39.186767    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:43:40.626786 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:43:40.626750    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:43:56.092143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.092109    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:43:56.096273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.096250    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.169996    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170051    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170134    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqbb\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170181    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170236    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170329    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170372    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170472    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.170706 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.170504    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.213456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.213377    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:43:56.271484 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271447    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271690 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271725    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271798 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271785    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271819    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqbb\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271869    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.271928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271908    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272030 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.271937    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272098 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272065    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272248 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272137    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272248 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272180    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272379 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272285    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272527    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.272717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.272697    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.274645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.274620    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.274842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.274824    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.327321 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.327285    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqbb\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.328437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.328415    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.411627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.411493    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:56.841533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.841497    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:43:56.847343 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:43:56.847302    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedee6deb_e0b4_49cc_89a4_a7ab88258786.slice/crio-2e6892651cd7eaab30e90f1778fd023260e8cff7da9c6d3e4eb9b53994d895f6 WatchSource:0}: Error finding container 2e6892651cd7eaab30e90f1778fd023260e8cff7da9c6d3e4eb9b53994d895f6: Status 404 returned error can't find the container with id 2e6892651cd7eaab30e90f1778fd023260e8cff7da9c6d3e4eb9b53994d895f6
Nov 01 22:43:56.849877 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.849842    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:56.849969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.849913    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:56.849969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:56.849942    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:43:57.289811 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.289712    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" event={"ID":"edee6deb-e0b4-49cc-89a4-a7ab88258786","Type":"ContainerStarted","Data":"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"}
Nov 01 22:43:57.289811 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.289785    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" event={"ID":"edee6deb-e0b4-49cc-89a4-a7ab88258786","Type":"ContainerStarted","Data":"2e6892651cd7eaab30e90f1778fd023260e8cff7da9c6d3e4eb9b53994d895f6"}
Nov 01 22:43:57.403802 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.403753    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podStartSLOduration=1.403738522 podStartE2EDuration="1.403738522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:57.401508605 +0000 UTC m=+1191.357548450" watchObservedRunningTime="2026-11-01 22:43:57.403738522 +0000 UTC m=+1191.359778360"
Nov 01 22:43:57.412737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.412705    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:43:57.414530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.414509    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:43:57.414619 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:57.414579    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:43:58.412684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:58.412652    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:43:58.413147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:58.412709    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:43:59.211237 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.211201    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:43:59.215455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.215432    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316189    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:43:59.316394 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316340    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316394 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316386    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316405    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316589    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316699    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hb9\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316830    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.316877 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.316847    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.317034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.317016    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.412375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.412338    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:43:59.412592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.412398    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:43:59.418700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.418666    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hb9\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.418714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.418916    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.418950    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419074    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419221    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419281    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419310    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419332    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419305    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419362    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.419943 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.419919    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.422260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.422238    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.435330 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.435281    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.497456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.497377    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hb9\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.497456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.497419    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:43:59.530423 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:43:59.530379    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:44:00.009380 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.009348    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:44:00.014566 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:44:00.014507    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8a8640_4e1f_40da_9a4a_50be556aa5b6.slice/crio-a7d92e4094b04ff468ecfccbc7b155479ae0420edea8f462677ef5f4cfc63630 WatchSource:0}: Error finding container a7d92e4094b04ff468ecfccbc7b155479ae0420edea8f462677ef5f4cfc63630: Status 404 returned error can't find the container with id a7d92e4094b04ff468ecfccbc7b155479ae0420edea8f462677ef5f4cfc63630
Nov 01 22:44:00.017181 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.017149    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:00.017255 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.017219    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:00.017255 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.017252    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:00.311924 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.311886    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" event={"ID":"df8a8640-4e1f-40da-9a4a-50be556aa5b6","Type":"ContainerStarted","Data":"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"}
Nov 01 22:44:00.312093 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.311951    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" event={"ID":"df8a8640-4e1f-40da-9a4a-50be556aa5b6","Type":"ContainerStarted","Data":"a7d92e4094b04ff468ecfccbc7b155479ae0420edea8f462677ef5f4cfc63630"}
Nov 01 22:44:00.412086 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.412052    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:44:00.412263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.412119    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:44:00.428372 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.428305    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podStartSLOduration=1.428284441 podStartE2EDuration="1.428284441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:00.424720975 +0000 UTC m=+1194.380760820" watchObservedRunningTime="2026-11-01 22:44:00.428284441 +0000 UTC m=+1194.384324286"
Nov 01 22:44:00.531522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:00.531478    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:44:01.412231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:01.412185    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.412518 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:01.412249    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:44:01.533580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:01.533520    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.534012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:01.533603    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:02.412313 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:02.412277    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.412512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:02.412330    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:44:02.531444 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:02.531409    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.531693 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:02.531468    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:03.412153 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:03.412110    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.412587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:03.412186    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 01 22:44:03.531236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:03.531195    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.531429 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:03.531257    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:04.522571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:04.522500    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:44:04.523118 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:04.523098    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:44:04.523794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:04.523771    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:44:04.531264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:04.531235    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:04.531432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:04.531310    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:05.531002 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:05.530965    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:05.531430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:05.531033    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:06.531269 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:06.531234    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 01 22:44:06.531724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:06.531288    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 01 22:44:07.918816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:07.918783    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:44:07.920156 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:07.920124    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:44:07.921284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:07.921265    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:44:10.501705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.501643    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.564668 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.564622    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.783624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.778648    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.819773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.819716    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:10.893249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.891832    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": context deadline exceeded"
Nov 01 22:44:10.893249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:10.892235    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.106616 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.106569    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.122726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.122680    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.151566 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:44:11.148399    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:44:11.168568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.165066    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.181720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.181649    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.351680 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.351620    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.505762 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.505640    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.514569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.506385    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:11.572576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.569984    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.572576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.570394    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:44:11.787638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.787004    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.787832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.787692    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:44:11.889672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.889625    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:11.919589 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.902253    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:11.939563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.918250    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:11.939925 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:11.939896    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:44:12.119777 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.119720    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.144837 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.128226    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.144837 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.130093    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:44:12.192948 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.192900    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.247694 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.208227    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.368568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.354646    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:12.826568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.773527    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:12.826568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.800827    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.901927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.901386    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:44:13.004590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.993985    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:13.004590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:12.994611    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:13.022579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.019030    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:13.159273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.159161    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:44:13.227532    5031 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.035s"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227639    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227691    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227723    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227750    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227775    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227802    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:13.229585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.227832    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:44:13.231716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.231104    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:13.237036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.232768    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:13.246668 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.242520    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:44:13.253685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.253297    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:44:13.253868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.253776    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:44:13.257943 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:13.257917    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:44:40.627619 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:44:40.627581    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:44:41.293059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.293027    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:44:41.297912 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.297881    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.339232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.339199    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:44:41.343565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.343448    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:44:41.350240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.350200    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.350873 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.350845    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:44:41.351350 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.351335    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.360124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.360091    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.368913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.368882    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.368920    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.368952    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369022    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369045    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369100    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369125    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369180    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369205    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369246    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369297    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369328    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jpb\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369385    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369421    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369447    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369476    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.369908 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369573    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.369908 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.369689    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnsh\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.400168 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.400127    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:44:41.404231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.404205    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.438999 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.438960    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471193    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471242    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcq7\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471279    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471356    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471389    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471413    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnsh\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471467    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471494    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471518    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471595    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.471635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471626    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471666    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471694    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471728    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471757    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471784    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471845    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471868    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471901    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471939    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jpb\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.471971    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472017    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472042    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472074    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472130    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472153    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472185    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8bb\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472236    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472294    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275c4\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472337    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472382    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.472420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472406    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472444    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472481    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472563    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472608    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472650    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472699    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472742    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472797    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472856    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472881    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.473489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.472911    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.474068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.473769    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.474834 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.474805    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.475304 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.475277    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.475932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.475903    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.476576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.476527    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:44:41.479202 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.479178    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.479442 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.479422    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.479953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.479894    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.481073 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.480610    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.481073 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.480873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.481073 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.481008    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.481962 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.481938    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.483502 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.483481    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.484036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.484014    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.485893 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.485330    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:44:41.486019 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.485997    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.492577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.490174    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:44:41.531249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.531213    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:44:41.534870 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.534831    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jpb\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.535424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.535397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnsh\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.544996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.543479    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.552164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.552130    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.574448 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574410    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.574448 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574448    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574484    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574516    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574611    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574641    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574682    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574726    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.574791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574749    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574803    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574827    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574862    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8bb\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574912    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574963    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275c4\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.574988    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575002    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575078    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575162    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575298    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575308    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575324    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcq7\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575355    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575404    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575444    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575472    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575579    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575295    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.575623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575616    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575715    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575743    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575781    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575814    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.575916    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.576203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.576127    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.576508 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.576222    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.576625 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.576599    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.576752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.576731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.578215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.577092    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.578215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.577121    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.578215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.577527    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.578215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.577593    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.578215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.577827    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.579959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.579934    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.582460 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.582434    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.584037 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.583984    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.584270 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.584244    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.584835 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.584816    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.586320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.586298    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.612838 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.612775    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:41.629069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.629008    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.630180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.630119    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.630180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.630166    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcq7\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.633082 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.633048    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275c4\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.633203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.633102    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8bb\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:41.634005 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.633985    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.669393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.669356    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:41.678958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.678923    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:41.691424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.691385    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:41.720247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:41.720208    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:42.050615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.049923    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:44:42.065718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.065676    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.065961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.065936    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.066117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.066096    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.124617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.123707    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:44:42.141532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.140970    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:44:42.162855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.162265    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.162855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.162359    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.162855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.162404    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.174592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.169167    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.174592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.169259    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.174592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.169304    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.186560 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.186510    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:44:42.201006 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.200972    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:44:42.205596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.204637    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.205596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.204768    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.205596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.204818    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.222064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.222021    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.224775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.224739    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.224975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.224953    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:44:42.521191 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.520031    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" event={"ID":"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12","Type":"ContainerStarted","Data":"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"}
Nov 01 22:44:42.521191 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.520098    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" event={"ID":"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12","Type":"ContainerStarted","Data":"5d1448bcf7b4829d560787cb7c161d659bc1ac9574f7b5a9e5d2aaa030d60378"}
Nov 01 22:44:42.525202 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.525169    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" event={"ID":"39489942-ba0d-45a9-90f9-5470817f18e4","Type":"ContainerStarted","Data":"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"}
Nov 01 22:44:42.525450 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.525433    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" event={"ID":"39489942-ba0d-45a9-90f9-5470817f18e4","Type":"ContainerStarted","Data":"6d34309786a9cac5edbcb56ae5c9fe20954429c8f61a4c90022bb7030d93b37e"}
Nov 01 22:44:42.536315 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.536272    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" event={"ID":"6db83d80-352b-4a54-93e1-427d5aadff08","Type":"ContainerStarted","Data":"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"}
Nov 01 22:44:42.538326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.536577    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" event={"ID":"6db83d80-352b-4a54-93e1-427d5aadff08","Type":"ContainerStarted","Data":"bf296c5945827f330cdd61472bb877497754a48e1d3b5991363923fe276d2f60"}
Nov 01 22:44:42.541118 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.541084    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" event={"ID":"185b08a2-1ec2-418f-90b1-43b584d46163","Type":"ContainerStarted","Data":"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"}
Nov 01 22:44:42.541332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.541140    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" event={"ID":"185b08a2-1ec2-418f-90b1-43b584d46163","Type":"ContainerStarted","Data":"d3f69ac03ea23178daaf3e22263a26b1c7dd3e2f48313b0e085ba118d1888fb1"}
Nov 01 22:44:42.543874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.543840    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" event={"ID":"ae11ef97-fb35-4928-9e06-091d73907b5f","Type":"ContainerStarted","Data":"eaf93156c3a895dc34d8bdaac0cfe40b6066df4948acedbe402c5476271682b8"}
Nov 01 22:44:42.544025 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.543898    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" event={"ID":"ae11ef97-fb35-4928-9e06-091d73907b5f","Type":"ContainerStarted","Data":"46d0026893d186b9fac3fe114f70b4e8842b9ca7c6a60952f3a9630d1d76d4bb"}
Nov 01 22:44:42.614824 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.614684    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:42.635091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.633982    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podStartSLOduration=1.633960147 podStartE2EDuration="1.633960147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.633070897 +0000 UTC m=+1236.589110764" watchObservedRunningTime="2026-11-01 22:44:42.633960147 +0000 UTC m=+1236.589999993"
Nov 01 22:44:42.684583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.683716    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:42.688562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.685812    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:42.694736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.693665    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:42.720718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.720687    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:42.753643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.753578    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podStartSLOduration=1.7535283929999999 podStartE2EDuration="1.753528393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.747608161 +0000 UTC m=+1236.703648007" watchObservedRunningTime="2026-11-01 22:44:42.753528393 +0000 UTC m=+1236.709568237"
Nov 01 22:44:42.887228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:42.887080    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podStartSLOduration=1.887052009 podStartE2EDuration="1.887052009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.869201054 +0000 UTC m=+1236.825240899" watchObservedRunningTime="2026-11-01 22:44:42.887052009 +0000 UTC m=+1236.843091855"
Nov 01 22:44:43.141479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.141350    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podStartSLOduration=2.141316597 podStartE2EDuration="2.141316597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.05294568 +0000 UTC m=+1237.008985525" watchObservedRunningTime="2026-11-01 22:44:43.141316597 +0000 UTC m=+1237.097356442"
Nov 01 22:44:43.259923 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.259846    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podStartSLOduration=2.259821346 podStartE2EDuration="2.259821346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.241666658 +0000 UTC m=+1237.197706504" watchObservedRunningTime="2026-11-01 22:44:43.259821346 +0000 UTC m=+1237.215861193"
Nov 01 22:44:43.668075 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.666554    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.668075 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.666624    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:43.672275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.672129    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.672275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.672210    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:43.683820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.683650    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.683820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.683732    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:43.695516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.695387    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.695516 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.695447    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:43.760722 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.760682    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.764694 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:43.764646    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:44.302937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.302886    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:44:44.305173 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.305130    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.342042 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.341997    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": context deadline exceeded"
Nov 01 22:44:44.351251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.351203    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.630694 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.629316    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.630694 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.629381    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:44.671070 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.671029    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.671755 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.671711    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:44.681585 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.681531    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.681874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.681846    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:44.696077 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.696038    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.696371 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.696343    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:44.722044 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.722003    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.722342 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.722305    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:44.810707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.810649    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.954755 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:44.954636    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.320613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.305209    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.320613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.305598    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.361109    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.361937    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.362107    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.362202    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.362209    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.362377    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.367533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.362593    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:44:45.393626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.370213    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.393626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.379060    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.641759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.641664    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.642054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.642005    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:45.682202 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.682160    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.682824 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.682772    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:45.722263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.696658    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.722263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.696724    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:45.722263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.722063    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.722263 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.722116    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:45.789731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.740680    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.789731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.740748    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:45.959630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.959486    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.991565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:45.991226    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:44:46.378645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.305702    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.652777    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.338327    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.381876    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.381906    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.383645    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.383681    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.383701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.434698    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" podUID="06afa0f3-960c-4468-8ad3-d0e6be05bc93" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.629426    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.662038    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.662955    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.663016    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:44:46.672627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.382606    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.705832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.678136    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:44:46.707458 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.681532    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.707721 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.707692    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.812658    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.812737    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.812838    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.812871    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.813450    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:46.819937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.813502    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:47.024678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:46.974116    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.430807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.380694    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:47.641753 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.641709    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.642019 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.641991    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:47.654743 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.654697    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.655976 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.655910    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.657205 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.657179    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.658647 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.658617    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.684621 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.664327    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.734733 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.734662    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.851566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.746783    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.863571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.859634    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:47.875568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.777759    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:44:47.940750 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.766779    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.032708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.751040    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.033071 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.033022    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:48.033243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.778181    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.033374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.033356    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:48.033501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:47.803820    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.033664 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.033644    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:48.034711 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.034673    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.338567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.338326    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.643626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.642936    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.643626 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.643017    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:48.678564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.675002    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.678564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.676622    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.678564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.676966    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.678564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.678030    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.704844 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.704795    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.705145 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.705100    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:48.714579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.714202    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.714579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.714308    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:48.719106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.719076    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.723774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.723718    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:48.733240 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.733201    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.733692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.733661    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:48.766985 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.766950    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.854863 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:48.854805    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.029745 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.029618    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.072521 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.070775    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.341705 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.341576    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.342337 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.342193    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:49.382280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.382251    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:49.623622 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.621348    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded"
Nov 01 22:44:49.678418 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.678390    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.678874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.678850    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:49.690532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.678702    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.693842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.690419    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.695209 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.678341    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.715978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.714777    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.715978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.714838    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:49.715978 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.714905    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Nov 01 22:44:49.719587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.716774    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.719587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.716828    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:49.747604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.742286    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.747604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.742353    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:49.747604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.743265    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.747604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.743308    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:49.747604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.743372    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.843629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.843282    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.860868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:49.860822    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.074565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.073645    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.074565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.074119    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.398736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.394118    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.616151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.616101    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.616583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.616504    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:50.673221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.673116    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.673518 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.673474    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:50.682773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.682736    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.683110 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.683071    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:50.691698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.691635    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.693078 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.693048    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.693287 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.693264    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:50.696673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.696634    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.697199 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.697165    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.724061 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.724023    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.724346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.724314    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:50.744734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.744668    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.858801 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.858733    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.868617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.867323    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.909643 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:50.904680    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.081566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.075650    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.081566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.076608    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.115565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.113461    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.163565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.163081    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:51.406758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.406645    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.420279 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.407487    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:51.632161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.632126    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.633073 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.633023    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:51.713406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.713293    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.715201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.713931    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.715437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.715410    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:51.715568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.714258    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.715710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.715688    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:51.715805 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.714909    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.715906 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.715891    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:51.716028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.713968    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.744587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.730680    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.744587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.730747    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:51.745740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.745659    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.873784 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.873723    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.881235 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:51.877935    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.092566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.082983    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.092566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.083401    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.110565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.110463    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.411619 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.407691    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.629568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.617614    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.629568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.617670    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:52.783668 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.662346    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.822718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.806868    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.822718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.807199    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.822718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.807399    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.921972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.921912    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.960567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.952965    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.964696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.964626    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:52.989566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.965019    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.000567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.992223    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:53.004565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:52.991158    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.008564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.004837    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:53.061635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.036382    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.071567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.064409    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.085761 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.085701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.086081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.085701    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.203752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.203710    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.276564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.251642    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:53.418629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.418492    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.707638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.707022    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.707638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.707085    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:53.707638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.707160    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.707638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.707183    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:53.707638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.707238    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.761750 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.761715    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:53.764174 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.763089    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.773258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.767590    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:53.799949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.763128    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.800258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.800225    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:53.800413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.763162    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.800531 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.800514    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 01 22:44:53.855108 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.855057    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.855762 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.855729    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.856182 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.856156    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.948724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:53.948672    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.079184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.079130    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.089935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.089891    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:44:54.090819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.090775    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.102233 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.102192    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:44:54.365146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.365059    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:44:54.420310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.418676    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:44:54.420310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.419584    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:44:54.489652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.489623    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:44:54.541811 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.541784    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:44:54.663293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.663166    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.663592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.663515    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 01 22:44:54.666358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.666324    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:44:54.671675 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.671467    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.673091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.671933    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 01 22:44:54.697186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.697152    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.697503 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.697444    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 22:44:54.718845 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.718815    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:44:54.735415 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.735375    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.735723 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.735689    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:54.751010 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.750979    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:44:54.765127 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:54.765096    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:44:55.657959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:55.657924    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:55.736567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:55.703409    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:55.736881 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:55.731263    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.764566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:55.740616    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 22:44:55.764858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:55.742921    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:44:56.251276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.251174    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:56.254492 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.254461    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:56.255367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.255334    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:56.257283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.256886    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:56.258127 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.257934    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:44:56.261144 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.261099    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:44:56.470682 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.470649    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:56.472083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.472051    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:56.481564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.476578    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:44:56.720707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.720670    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:56.725206 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:56.725183    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:44:57.467588 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:44:57.467525    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:45:02.777658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.777619    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"]
Nov 01 22:45:02.782229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.782200    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.832305 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.832269    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.832494 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.832399    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.832582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.832508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f724p\" (UniqueName: \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.882633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.882599    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:02.882633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.882624    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:02.888023 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.887990    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"]
Nov 01 22:45:02.933489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.933449    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.933711 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.933570    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.933711 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.933616    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f724p\" (UniqueName: \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.934197 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.934171    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.936241 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.936215    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:02.988317 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:02.988282    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f724p\" (UniqueName: \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\") pod \"maas-api-revoke-sub-15ae82c8-jh68r\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:03.097887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:03.097850    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:03.477559 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:03.477452    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"]
Nov 01 22:45:03.482682 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:45:03.482653    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48438123_3ba2_44b4_90fd_f259922bdca4.slice/crio-18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030 WatchSource:0}: Error finding container 18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030: Status 404 returned error can't find the container with id 18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030
Nov 01 22:45:03.485213 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:03.485197    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:45:03.509326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:03.509294    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r" event={"ID":"48438123-3ba2-44b4-90fd-f259922bdca4","Type":"ContainerStarted","Data":"18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030"}
Nov 01 22:45:08.547353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:08.547313    5031 generic.go:356] "Generic (PLEG): container finished" podID="48438123-3ba2-44b4-90fd-f259922bdca4" containerID="bbf4f9d5c003f8183b952092c5cca633cd90eb92cd15f74b950fe121c5fb36da" exitCode=0
Nov 01 22:45:08.547799 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:08.547393    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r" event={"ID":"48438123-3ba2-44b4-90fd-f259922bdca4","Type":"ContainerDied","Data":"bbf4f9d5c003f8183b952092c5cca633cd90eb92cd15f74b950fe121c5fb36da"}
Nov 01 22:45:09.733748 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.733717    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:09.840494 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.840396    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\") pod \"48438123-3ba2-44b4-90fd-f259922bdca4\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") "
Nov 01 22:45:09.840696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.840511    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\") pod \"48438123-3ba2-44b4-90fd-f259922bdca4\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") "
Nov 01 22:45:09.840753 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.840728    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\" (UniqueName: \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\") pod \"48438123-3ba2-44b4-90fd-f259922bdca4\" (UID: \"48438123-3ba2-44b4-90fd-f259922bdca4\") "
Nov 01 22:45:09.840933 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.840905    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca" pod "48438123-3ba2-44b4-90fd-f259922bdca4" (UID: "48438123-3ba2-44b4-90fd-f259922bdca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:09.841254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.841233    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-openshift-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:09.843026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.842995    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p" pod "48438123-3ba2-44b4-90fd-f259922bdca4" (UID: "48438123-3ba2-44b4-90fd-f259922bdca4"). InnerVolumeSpecName "kube-api-access-f724p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:09.843135 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.843034    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca" pod "48438123-3ba2-44b4-90fd-f259922bdca4" (UID: "48438123-3ba2-44b4-90fd-f259922bdca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:09.942453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.942414    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f724p\" (UniqueName: \"kubernetes.io/projected/48438123-3ba2-44b4-90fd-f259922bdca4-kube-api-access-f724p\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:09.942453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:09.942446    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48438123-3ba2-44b4-90fd-f259922bdca4-maas-api-xks-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:10.561847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:10.561807    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"
Nov 01 22:45:10.561847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:10.561836    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r" event={"ID":"48438123-3ba2-44b4-90fd-f259922bdca4","Type":"ContainerDied","Data":"18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030"}
Nov 01 22:45:10.562061 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:10.561886    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b77a8fbac4b8be0111baddb353f5d2ce6499c8c70d670af6f8b0b6b0e04030"
Nov 01 22:45:10.627338 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:45:10.627303    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:45:11.698569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.698518    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:45:11.699118 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.699105    5031 cpu_manager.go:414] "containerMap: removing container" podUID="48438123-3ba2-44b4-90fd-f259922bdca4" containerName="revoke-keys"
Nov 01 22:45:11.699161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.699120    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48438123-3ba2-44b4-90fd-f259922bdca4" containerName="revoke-keys"
Nov 01 22:45:11.699232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.699222    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="48438123-3ba2-44b4-90fd-f259922bdca4" containerName="revoke-keys"
Nov 01 22:45:11.702821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.702802    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.759109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.759074    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:45:11.763203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.763178    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.770082 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770046    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770221    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770330 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770308    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770445    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770769    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tl2\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770906 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770850    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.770964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.770902    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.771076 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.771051    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.771166 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.771149    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.805562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.805406    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:45:11.829687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.829650    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:45:11.873139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873098    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfskg\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873451 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873421    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873498    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.873596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873587    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873631    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873713    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873827 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873808    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tl2\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.873911 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873893    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.873962 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.873947    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874023    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874060    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.874101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874089    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.874214 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874197    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874303 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874286    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874371 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874318    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874371 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874366    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874420    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874440    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874470    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874703    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874835    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874902    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.874939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.874942    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.877192 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.877170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.877281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.877223    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.927892 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.927847    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.928074 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.927959    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tl2\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:11.975883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.975795    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.975883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.975857    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.975883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.975875    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.975986    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976016    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfskg\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976054    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976130    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976172    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976202    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976267    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976690 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976663    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976716    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.976775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.976729    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.977229 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.977200    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.978812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.978788    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:11.979169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:11.978980    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:12.016976 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.016942    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:12.030715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.030687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:12.030918 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.030897    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfskg\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:12.079608 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.079577    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:12.454038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.453994    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:45:12.458463 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:45:12.458411    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90d7a16_3248_4931_bb95_9f87511cbc17.slice/crio-8087e3dadc6ac2d9ba248a3870a08bf25b4796ba52c25938ac0d4eb034d1eb53 WatchSource:0}: Error finding container 8087e3dadc6ac2d9ba248a3870a08bf25b4796ba52c25938ac0d4eb034d1eb53: Status 404 returned error can't find the container with id 8087e3dadc6ac2d9ba248a3870a08bf25b4796ba52c25938ac0d4eb034d1eb53
Nov 01 22:45:12.461828 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.461786    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.461931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.461862    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.461931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.461898    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.517083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.515449    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:45:12.525179 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.525137    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.525326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.525217    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.525326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.525262    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:12.580689 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.580655    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" event={"ID":"b90d7a16-3248-4931-bb95-9f87511cbc17","Type":"ContainerStarted","Data":"7aba0af6bd37810833fb0916113b2f27aacab2e00cf176fe55bf06b09ffdde59"}
Nov 01 22:45:12.580878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.580720    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" event={"ID":"b90d7a16-3248-4931-bb95-9f87511cbc17","Type":"ContainerStarted","Data":"8087e3dadc6ac2d9ba248a3870a08bf25b4796ba52c25938ac0d4eb034d1eb53"}
Nov 01 22:45:12.582606 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.582577    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" event={"ID":"499797d4-f4ee-4657-b153-62e754ac928f","Type":"ContainerStarted","Data":"b489a42310331d56b607042bcc089befea4545cf28cf4138d982ca3ef4772a51"}
Nov 01 22:45:12.698465 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:12.698383    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podStartSLOduration=1.698364183 podStartE2EDuration="1.698364183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:12.69734694 +0000 UTC m=+1266.653386782" watchObservedRunningTime="2026-11-01 22:45:12.698364183 +0000 UTC m=+1266.654404031"
Nov 01 22:45:13.017873 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:13.017785    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:13.019409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:13.019366    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:13.019590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:13.019425    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:13.611235 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:13.611180    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" event={"ID":"499797d4-f4ee-4657-b153-62e754ac928f","Type":"ContainerStarted","Data":"89aaf269fc703f1a44096f9c1558871dd5e8f98a1520bec617127214a36d88d7"}
Nov 01 22:45:13.783566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:13.782730    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podStartSLOduration=2.7827116629999997 podStartE2EDuration="2.782711663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:13.782054655 +0000 UTC m=+1267.738094512" watchObservedRunningTime="2026-11-01 22:45:13.782711663 +0000 UTC m=+1267.738751508"
Nov 01 22:45:14.018293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:14.018205    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:14.018293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:14.018267    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:14.082222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:14.081967    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:14.082222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:14.082109    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:14.082222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:14.082154    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:15.017464 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:15.017411    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:15.017719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:15.017485    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:15.085593 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:15.085556    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:15.086179 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:15.086147    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:16.017637 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:16.017533    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:16.017637 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:16.017602    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:16.089565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:16.081949    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:16.089565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:16.082016    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:17.017511 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:17.017473    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:17.017749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:17.017588    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:17.101567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:17.080126    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:17.101567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:17.080190    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:18.018312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:18.018267    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:18.018569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:18.018349    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:18.083034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:18.082890    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:18.083034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:18.082970    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:19.018446 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:19.018402    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:45:19.018872 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:19.018495    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:45:19.080388 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:19.080350    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Nov 01 22:45:19.080601 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:19.080435    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Nov 01 22:45:20.077990 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.077948    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:20.080438 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.079754    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:20.083294 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.083226    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:45:20.190690 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.190577    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:20.191812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.191789    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:20.192461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:20.192440    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:45:24.657561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:24.656637    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:24.681566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:24.675726    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:45:24.721158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:24.720656    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:24.814678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:24.814623    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:24.925764 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:24.925648    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.084741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.084689    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.310794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.310745    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.469719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.469673    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.490708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.490650    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.664761 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.664646    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.665854 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.665823    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:45:25.685221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.679379    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.685221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.679859    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:45:25.723735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.723684    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:45:25.724379 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.724260    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:45:25.855691 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.855132    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.953411 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:25.953309    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.154501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.054454    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.155435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.155410    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:45:26.183652 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.183598    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.476567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.473476    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.494028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.493985    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.729105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.728998    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.753672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.753612    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.801633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.763667    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.968568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.946266    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.991389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:26.991264    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:45:27.025568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.023772    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:45:27.025568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.023845    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:45:27.025568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.023885    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:45:27.028208 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.028146    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:45:27.165569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.161331    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.165569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.161836    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.204568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.165986    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy" containerID="cri-o://19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214" gracePeriod=30
Nov 01 22:45:27.204568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.186971    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:45:27.207684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.207648    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.312810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.312658    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:45:27.312810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.312725    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.516737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.516670    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.590624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.565644    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.626772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.626721    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:45:27.627060 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.627029    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.725903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.715017    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:45:27.725903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.716473    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.993623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:27.993232    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.026715 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.026658    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.031623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.027433    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.036666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.036616    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.214921 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.214876    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.240565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.237686    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.240565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.239583    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.544569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.542832    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.588740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.588692    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:45:28.626561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.589011    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.626561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:28.605642    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.029018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.028915    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.054875 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.054818    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.073571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.072900    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.083644 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.074803    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.232660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.232609    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.238730 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.238683    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.273990 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.242763    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.567565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.544797    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.654572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.545787    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.654937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.633042    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.969854 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:29.969742    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded"
Nov 01 22:45:30.139568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.137287    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.143014 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.142972    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.165568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.145877    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.262109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.261974    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.378972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.169703    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.619579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.558720    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.683561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.629938    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.793459 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.788996    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.829221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.829189    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:45:30.829415 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.829260    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:45:30.829415 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.829298    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:45:30.829657 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.829630    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:45:30.830188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.830171    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:45:30.834361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.834329    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:45:30.834517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.834373    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:45:30.834517 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.834404    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:45:30.835513 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:30.835491    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:45:32.803462 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.803432    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:45:32.824334 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.824290    5031 generic.go:356] "Generic (PLEG): container finished" podID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerID="19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214" exitCode=0
Nov 01 22:45:32.824576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.824375    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"
Nov 01 22:45:32.824576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.824454    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" event={"ID":"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08","Type":"ContainerDied","Data":"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"}
Nov 01 22:45:32.824576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.824531    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp" event={"ID":"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08","Type":"ContainerDied","Data":"5ef214b065d2c7f780f3c669429cef5e53692f594e9ae068a4870b265930c262"}
Nov 01 22:45:32.824576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.824570    5031 scope.go:122] "RemoveContainer" containerID="19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"
Nov 01 22:45:32.836032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.836008    5031 scope.go:122] "RemoveContainer" containerID="19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"
Nov 01 22:45:32.836377 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:45:32.836352    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214\": container with ID starting with 19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214 not found: ID does not exist" containerID="19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"
Nov 01 22:45:32.836441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.836380    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214"} err="failed to get container status \"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214\": rpc error: code = NotFound desc = could not find container \"19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214\": container with ID starting with 19086ddd93036c57038de7a1c9405a21dedae9b5d83b8da43f5cbf37d9683214 not found: ID does not exist"
Nov 01 22:45:32.931178 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931140    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931372 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931227    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931372 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931320    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931507 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931420    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931507 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931472    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931601    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931657    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931799 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931719    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931799 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\") pod \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\" (UID: \"62b26e64-e1e1-4e85-8ab9-abac4ddb2f08\") "
Nov 01 22:45:32.931905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931649    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:32.931961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.931924    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:32.932214 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932192    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:32.932528 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932493    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:32.932872 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932849    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.932954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932879    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.932954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932894    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.932954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.932908    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:32.933409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.933384    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:32.934331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.934307    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:32.935621 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.935591    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:32.935718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.935700    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "kube-api-access-q9hqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:32.935843 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:32.935820    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token" pod "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" (UID: "62b26e64-e1e1-4e85-8ab9-abac4ddb2f08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.034578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.034478    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hqm\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-kube-api-access-q9hqm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.034578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.034514    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.034578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.034524    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.034578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.034533    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.034578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.034559    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.243043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.243005    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:45:33.248006 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:33.247973    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hbrrp"]
Nov 01 22:45:34.172956 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:34.172916    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" path="/var/lib/kubelet/pods/62b26e64-e1e1-4e85-8ab9-abac4ddb2f08/volumes"
Nov 01 22:45:40.628492 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:45:40.628446    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:45:45.170180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.170138    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:45:45.170783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.170768    5031 cpu_manager.go:414] "containerMap: removing container" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy"
Nov 01 22:45:45.170830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.170785    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy"
Nov 01 22:45:45.170878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.170869    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b26e64-e1e1-4e85-8ab9-abac4ddb2f08" containerName="istio-proxy"
Nov 01 22:45:45.178318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.178291    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.236608 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.236564    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:45:45.293376 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293333    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293640 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293475    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293640 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293617    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293693    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293804 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293783    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293900 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293831    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.293971 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293899    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxs2c\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.294027 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.293977    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.294105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.294082    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.396891 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.396850    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397096 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.396928    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397096 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397011    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397219 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397101    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397219 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397141    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397219 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397179    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397252    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397290    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397298    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397339    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397366    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxs2c\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397602    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.397685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.397622    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.398139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.398121    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.399525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.399501    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.399790 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.399768    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.457851 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.457766    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.458297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.458277    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxs2c\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.492994 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.492959    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:45.947398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.947366    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:45:45.954869 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.954824    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:45.954991 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.954921    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:45.955032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:45.955002    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:46.928493 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:46.928456    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" event={"ID":"448a2604-d3d3-4ff1-b096-7ee993f98dcb","Type":"ContainerStarted","Data":"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"}
Nov 01 22:45:46.928960 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:46.928517    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" event={"ID":"448a2604-d3d3-4ff1-b096-7ee993f98dcb","Type":"ContainerStarted","Data":"15d0e4378c34b1112eb43404b22b9aa62e5276e54128ec86c827625a036cca67"}
Nov 01 22:45:47.174564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:47.172226    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podStartSLOduration=2.172202816 podStartE2EDuration="2.172202816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:47.111746916 +0000 UTC m=+1301.067786764" watchObservedRunningTime="2026-11-01 22:45:47.172202816 +0000 UTC m=+1301.128242662"
Nov 01 22:45:47.493335 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:47.493292    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:47.495325 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:47.495299    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:47.495397 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:47.495352    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:48.493362 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:48.493322    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:48.493816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:48.493390    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:49.493602 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:49.493568    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:49.494053 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:49.493627    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:50.493863 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:50.493825    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:50.494265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:50.493897    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:51.494021 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:51.493977    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:51.494423 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:51.494045    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:52.493632 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:52.493591    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 22:45:52.493815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:52.493674    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 22:45:53.737389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:53.737340    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:53.737824 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:53.737772    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:53.738683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:53.738656    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:45:57.031794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.031757    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:45:57.037336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.037312    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.115841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.112194    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:45:57.147644 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.147605    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.158023 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.157976    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158192 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158071    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158291 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158272    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158354 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158337    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158376    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h46\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158433 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158419    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158469 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158459    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158548 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158518    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.158604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.158579    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.195173 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.195138    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:45:57.220277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.220241    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:45:57.261107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261063    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.261375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261354    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.261501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261488    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.261620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261605    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h46\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.261727 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.261820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261808    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.261905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261895    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.261987 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.261973    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.262083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262072    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldnx\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.262187 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262175    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.262270 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262259    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.262385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262374    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.262473 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262462    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.262623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262608    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.262881 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.262857    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.263024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.263013    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.263105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.263094    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.263351 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.263338    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.264026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.264003    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.267296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.267268    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.268348 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.268321    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.269345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.269322    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.269578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.269390    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.269719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.269639    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.280033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.279994    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.324513 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.324485    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h46\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.331159 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.331127    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.369556    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376046    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376107    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376150    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376187    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldnx\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376251    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376279    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376321    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376446    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.376983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.376620    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.380634 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.380485    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.385618    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.385904    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.387871    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.389047    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.389133    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.391688 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.389301    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.446388 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.446352    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldnx\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.449913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.449879    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.531563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.523814    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:57.853708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.853671    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:45:57.869082 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:45:57.868985    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4b1daa_bf3e_403d_845b_7558ca6165be.slice/crio-076caf81e5935a6887152b4f83b68d5dd988f89786e2316b31ca7c0763365039 WatchSource:0}: Error finding container 076caf81e5935a6887152b4f83b68d5dd988f89786e2316b31ca7c0763365039: Status 404 returned error can't find the container with id 076caf81e5935a6887152b4f83b68d5dd988f89786e2316b31ca7c0763365039
Nov 01 22:45:57.876566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.876482    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:57.876691 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.876598    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:57.876691 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.876639    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:57.979902 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.979871    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:45:57.985214 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:45:57.985138    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ddc310_ff4a_407e_b55b_304ffb3fd629.slice/crio-fa567b02075bce128ce1fbfdb2431ec47d42bc8bf69ba41feea699335b301160 WatchSource:0}: Error finding container fa567b02075bce128ce1fbfdb2431ec47d42bc8bf69ba41feea699335b301160: Status 404 returned error can't find the container with id fa567b02075bce128ce1fbfdb2431ec47d42bc8bf69ba41feea699335b301160
Nov 01 22:45:57.988952 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.988905    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:57.989054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.989014    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:57.989112 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:57.989058    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:45:58.268859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.268767    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" event={"ID":"fb4b1daa-bf3e-403d-845b-7558ca6165be","Type":"ContainerStarted","Data":"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"}
Nov 01 22:45:58.268859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.268836    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" event={"ID":"fb4b1daa-bf3e-403d-845b-7558ca6165be","Type":"ContainerStarted","Data":"076caf81e5935a6887152b4f83b68d5dd988f89786e2316b31ca7c0763365039"}
Nov 01 22:45:58.271480 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.271446    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" event={"ID":"d9ddc310-ff4a-407e-b55b-304ffb3fd629","Type":"ContainerStarted","Data":"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"}
Nov 01 22:45:58.271687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.271513    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" event={"ID":"d9ddc310-ff4a-407e-b55b-304ffb3fd629","Type":"ContainerStarted","Data":"fa567b02075bce128ce1fbfdb2431ec47d42bc8bf69ba41feea699335b301160"}
Nov 01 22:45:58.374434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.374403    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:45:58.376601 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.376571    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:45:58.376721 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.376633    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:45:58.403613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.403557    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podStartSLOduration=2.403525116 podStartE2EDuration="2.403525116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:58.39974595 +0000 UTC m=+1312.355785795" watchObservedRunningTime="2026-11-01 22:45:58.403525116 +0000 UTC m=+1312.359564960"
Nov 01 22:45:58.525122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.525021    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:45:58.526386 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:58.526339    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podStartSLOduration=2.526327157 podStartE2EDuration="2.526327157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:58.523327145 +0000 UTC m=+1312.479366989" watchObservedRunningTime="2026-11-01 22:45:58.526327157 +0000 UTC m=+1312.482367000"
Nov 01 22:45:59.378567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:59.376666    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:45:59.378567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:59.376732    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:45:59.538174 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:59.534652    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:45:59.538174 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:45:59.534712    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:00.375037 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:00.374998    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:00.375314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:00.375283    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:00.525003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:00.524972    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:00.525399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:00.525025    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:01.400470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:01.399953    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:01.400470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:01.400037    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:01.525054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:01.525009    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:01.525454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:01.525077    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:02.375564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:02.375056    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:02.375564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:02.375134    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:02.528568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:02.528167    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:02.528568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:02.528330    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:03.141231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.097533    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.141231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.128386    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:03.257642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.237850    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.297710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.297661    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.399628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.393575    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:03.399628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.393638    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:03.561566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.555141    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.600821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.600688    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:03.600821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.600756    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:03.649580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.649155    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.783761 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.783650    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.884168 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:03.884129    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.100697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.100649    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.130065 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.130028    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:46:04.164575 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.134243    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.168966 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.164919    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.313903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.313859    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.344569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.319746    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.345795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.345755    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": context deadline exceeded"
Nov 01 22:46:04.421655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.391071    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:04.421655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.391146    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:04.609614 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.531949    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:46:04.786914 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.786818    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:46:04.787096 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.786933    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:46:04.787365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.730638    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.787709 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.613710    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:04.787901 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.787874    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:04.788102 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.730772    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.788665 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.788646    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:46:04.788878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.788865    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:46:04.856528 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.856489    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Nov 01 22:46:04.857750 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.857716    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.858345 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.858324    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:46:04.911566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.889240    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.915559 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:04.911967    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:46:05.189566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.172480    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.214831 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.196677    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.243579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.219878    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:46:05.243579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.228828    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.348092 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.344041    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.351008 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.349664    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.352408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.349720    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.400169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.391734    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:05.400169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.391795    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:05.588567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.584662    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:05.588567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.584734    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.634513 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.600184    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:05.634513 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.600254    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:05.792332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.789210    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.792332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.789495    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.865197 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.863080    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.905930 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:05.905870    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.141841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.141714    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.142146 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.142097    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.321567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.310736    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.381568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.326616    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.381568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.353781    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.381568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.354648    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.381568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.361208    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.381568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.361257    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.437623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.434464    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.456579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.434776    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:06.456579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.449762    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:06.456579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.434922    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" podUID="06afa0f3-960c-4468-8ad3-d0e6be05bc93" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.456579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.434876    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.456579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.449860    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.578836 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.578790    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:06.579131 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.579097    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:06.817532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.817474    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.844567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.819195    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.943686 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.943623    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.968567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:06.965271    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.315507 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.315459    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.355604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.329944    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.357595 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.357533    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.371628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.359038    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.443910 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.442942    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.454114 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.454078    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:07.454381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.454357    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:07.530577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.530520    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:07.530863 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:07.530830    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:08.044330 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.044270    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.045118 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.045086    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.045960 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.045932    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.048157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.048112    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.356629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.341293    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.496974 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.368989    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.501582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.369029    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.501582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.369049    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.501582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.389043    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:08.501582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.498018    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:08.501582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.477213    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.566625 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.550835    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:08.601803 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:08.601755    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:09.049600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.049529    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.160758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.091973    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.181950 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.107043    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.197562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.107223    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.225937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.225882    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:09.226279 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.226246    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.448132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.437360    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:09.448132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.437447    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:09.529566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.500949    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.529566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.501385    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.529566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.501714    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.529566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.501979    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.529566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.503807    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.561717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.561679    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:09.562031 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.561988    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:09.568169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.568125    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.624794 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.624755    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:09.625136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:09.625103    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.021217 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.020931    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.082561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.082462    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.085666 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.085627    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:10.085952 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.085920    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.222572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.201612    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.222572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.202026    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.222572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.202287    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.467852 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.467817    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:10.468133 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.468107    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:10.551578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.541915    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:10.551578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.541977    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:10.551578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.542070    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.551578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.542411    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.574583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.557853    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.574583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.558685    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.586578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.574974    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.984563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:10.954571    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.085339 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.085292    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.093026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.092904    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.257630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.210809    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:11.257630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.211110    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:11.257630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.211367    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": context deadline exceeded"
Nov 01 22:46:11.555046 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.555008    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.555235 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.555008    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.570576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.561213    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:11.570912 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.570872    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:11.571043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.564592    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:11.638572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.564555    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.672700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.610146    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:11.688740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.688690    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:11.692777 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.638915    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:11.732817 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.732775    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:11.736664 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:11.736621    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.092567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.090687    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.143565 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:46:12.093636    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:46:12.143565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.095406    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.211835 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.211790    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.213087 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.213053    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:12.239441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.224196    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": context deadline exceeded"
Nov 01 22:46:12.398921 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.398739    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:12.399224 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.399172    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.399928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.399904    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:12.400117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.400094    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:12.400385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.400356    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:46:12.541638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.535467    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:46:12.573566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.570257    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:12.598295 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.598267    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:12.598581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.598189    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.599935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.598140    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:12.636163 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.598210    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded"
Nov 01 22:46:12.688582 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.673043    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:12.744565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.736601    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:12.902941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.902907    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:46:12.943700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:12.942641    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.157681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.157631    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.215766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.215661    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.270578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.269531    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:46:13.282610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.281661    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.322268 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.321712    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.391766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.390287    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 01 22:46:13.391766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.390350    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 01 22:46:13.577599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.556517    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 22:46:13.577599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.556604    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 22:46:13.604580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.603679    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.640569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.640487    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.679707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.676639    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:13.748934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:13.748777    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded"
Nov 01 22:46:14.087611 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.051978    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:46:14.097078 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.094739    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:46:14.120283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.120248    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:46:14.170758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.170719    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:46:14.262751 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.261293    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 22:46:14.274563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.273142    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:46:14.274884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.274857    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:46:14.566361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.566313    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:46:14.575054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.574916    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:46:14.641697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.640007    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:46:14.642222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:14.642193    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:46:22.105831 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.105789    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 22:46:22.111835 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.111813    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.165630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.165595    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 22:46:22.165827 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.165653    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 22:46:22.165827 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.165689    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 22:46:22.209509 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.209476    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b55rk\""
Nov 01 22:46:22.214275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.214247    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 22:46:22.245318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245275    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.245526 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245441    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.245526 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245508    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.245718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245698    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.245845 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245820    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdh2\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.245917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245900    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.246001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.245982    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.347483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.347448    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.347704 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.347501    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.347753 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.347721    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.347806 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.347791    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.348103 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.348049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.348264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.348182    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.348347 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.348288    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdh2\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.348413 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.348343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.350201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.350175    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.350518 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.350498    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.350695 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.350681    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.350889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.350867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.406026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.405940    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.407409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.407388    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdh2\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vrjtc\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.425052 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.424996    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:22.868754 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:22.868723    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 22:46:22.872574 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:46:22.872497    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabeed4b_35e8_4266_88c1_6bbec4c22cd8.slice/crio-93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9 WatchSource:0}: Error finding container 93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9: Status 404 returned error can't find the container with id 93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9
Nov 01 22:46:23.427555 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:23.427498    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" event={"ID":"cabeed4b-35e8-4266-88c1-6bbec4c22cd8","Type":"ContainerStarted","Data":"93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9"}
Nov 01 22:46:26.909723 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:26.909672    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:46:26.910051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:26.909765    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:46:27.462067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:27.462024    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" event={"ID":"cabeed4b-35e8-4266-88c1-6bbec4c22cd8","Type":"ContainerStarted","Data":"7e508a471ca251ad42c931a12616e97cc4f9ef277625b799ee539badba75c8d0"}
Nov 01 22:46:27.463579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:27.463299    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:27.465029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:27.465000    5031 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vrjtc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:46:27.465143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:27.465091    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:46:27.579677 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:27.579596    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" podStartSLOduration=1.546368491 podStartE2EDuration="5.579579331s" totalImagesPullingTime="4.03321084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:27.576125842 +0000 UTC m=+1341.532165690" watchObservedRunningTime="2026-11-01 22:46:27.579579331 +0000 UTC m=+1341.535619176"
Nov 01 22:46:28.469495 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:28.469455    5031 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vrjtc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:46:28.469898 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:28.469525    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:46:29.476437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:29.476404    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 22:46:40.626975 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:46:40.626934    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:46:42.464077 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.464018    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 22:46:42.474899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.472966    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.580714 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580669    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2nc\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.580888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580741    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.580888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580780    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.580888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580861    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.580888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580885    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.581161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.580915    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.581161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.581032    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.581161 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.581101    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.581317 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.581171    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.682564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.682501    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.682927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.682900    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2nc\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.683141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.683123    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.683279 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.683264    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.683479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.683462    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.683673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.683654    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.683820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.683804    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.684098 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.684074    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.684738    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.684997    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.685020    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.685063    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.685217    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.685901 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.685880    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.688327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.688301    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.688839 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.688773    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.697525 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.697484    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 22:46:42.745756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.745676    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.754136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.754098    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2nc\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-7pbvq\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:42.799914 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:42.799870    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:44.470568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.418196    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.470568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.456752    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.717955 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.484339    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:44.718571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.496124    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.718697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.496177    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.721051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.652353    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.775736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.758123    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" podUID="06afa0f3-960c-4468-8ad3-d0e6be05bc93" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.18:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.827568 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.826679    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Nov 01 22:46:44.868761 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.868709    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.885310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.885260    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.938592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.908940    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.980627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:44.955994    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 22:46:45.087571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.074386    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:46:45.130567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.129843    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:46:45.130567 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.129929    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:46:45.184693 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.184641    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.317667 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.317623    5031 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.682665 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.672690    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:45.682665 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.672779    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.708726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.708689    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:45.708981 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.708951    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.729787 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.729747    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.731123 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.730030    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.775348 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.730636    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 01 22:46:45.776779 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.730679    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.777181 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.735127    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:46:45.777288 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.777260    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:46:45.777358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.777309    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:46:45.777358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.777349    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:46:45.899319 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.899210    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.540577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:45.906662    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx" podUID="0d80334c-9fa2-413b-94ea-a4c353344f29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.540577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.031959    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.549573 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.199104    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.379996    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" event={"ID":"ea484b4f-95ce-499c-9c25-1ec54cc3967f","Type":"ContainerStarted","Data":"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"}
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.562942    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.562974    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" event={"ID":"ea484b4f-95ce-499c-9c25-1ec54cc3967f","Type":"ContainerStarted","Data":"f99907c800c3c420f9c7ba2e6c96ad08d5c43ae83ded1bef3e3ade2c904f35f9"}
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.563035    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.563073    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.563107    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:46:46.571678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.432663    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.606580 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.453006    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-hxwjj" podUID="06afa0f3-960c-4468-8ad3-d0e6be05bc93" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.18:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:46.746401 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.726117    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 22:46:46.751477 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.747313    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 01 22:46:46.811499 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.811465    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:46.812058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.812021    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:46.825253 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.822840    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 22:46:46.825253 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.822890    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:46.830382 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.828301    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 22:46:46.910662 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:46.882737    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 22:46:47.028724 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.028691    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx"
Nov 01 22:46:47.073123 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.073035    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 22:46:47.078572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.075261    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 22:46:47.186566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.179331    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 22:46:47.303365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.303295    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podStartSLOduration=5.303269355 podStartE2EDuration="5.303269355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:47.276351711 +0000 UTC m=+1361.232391556" watchObservedRunningTime="2026-11-01 22:46:47.303269355 +0000 UTC m=+1361.259309228"
Nov 01 22:46:47.801891 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.801854    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:47.802467 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:47.802436    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:48.801190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:48.801151    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:48.801402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:48.801238    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:49.801264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:49.801224    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:49.801721 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:49.801296    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:50.801130 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:50.801092    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:50.801332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:50.801162    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:51.800945 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:51.800907    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:51.801140 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:51.800962    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:52.801010 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:52.800967    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:52.801416 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:52.801023    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:53.800878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:53.800834    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 01 22:46:53.801143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:53.800925    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 01 22:46:55.707673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:55.707626    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:55.708144 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:55.708027    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:46:55.709025 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:46:55.709001    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 22:47:10.627476 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:47:10.627433    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:47:40.629296 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:47:40.629255    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:48:10.627471 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:48:10.627429    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:48:40.626878 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:48:40.626787    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:48:49.326238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:48:49.325680    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-xvntw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:48:49.326238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:48:49.325804    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:49:01.897314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:01.897279    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:49:01.897774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:01.897497    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" containerID="cri-o://aeb12efbfff8b3cadc3cf1a2e5b145411b7c211efbeafe98b2ff5136fffdddd7" gracePeriod=30
Nov 01 22:49:06.862122 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:06.862087    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:49:06.862533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:06.862150    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:49:07.177281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.177254    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:49:07.202769 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.202735    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.202945 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.202816    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.202945 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.202893    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.202949    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203048    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203128    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203223 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203049    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.203353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203234    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.203353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203291    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203328    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:49:07.203512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203358    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203406    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\") pod \"caef116d-b8ec-4816-96a0-c69f81eb2f62\" (UID: \"caef116d-b8ec-4816-96a0-c69f81eb2f62\") "
Nov 01 22:49:07.203512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203430    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.203888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.203743    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.204145 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.204122    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.204256 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.204191    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caef116d-b8ec-4816-96a0-c69f81eb2f62-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.204337 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.204322    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.204399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.204342    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.204399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.204356    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.205490 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.205467    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.205620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.205483    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:49:07.205620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.205511    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "kube-api-access-tb28h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:49:07.205730 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.205714    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo" pod "caef116d-b8ec-4816-96a0-c69f81eb2f62" (UID: "caef116d-b8ec-4816-96a0-c69f81eb2f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:49:07.305610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.305574    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.305610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.305604    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.305610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.305614    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caef116d-b8ec-4816-96a0-c69f81eb2f62-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.305847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.305624    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb28h\" (UniqueName: \"kubernetes.io/projected/caef116d-b8ec-4816-96a0-c69f81eb2f62-kube-api-access-tb28h\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.470773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.470688    5031 scope.go:122] "RemoveContainer" containerID="aeb12efbfff8b3cadc3cf1a2e5b145411b7c211efbeafe98b2ff5136fffdddd7"
Nov 01 22:49:07.652676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.652645    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"
Nov 01 22:49:07.652866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.652636    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" event={"ID":"caef116d-b8ec-4816-96a0-c69f81eb2f62","Type":"ContainerDied","Data":"aeb12efbfff8b3cadc3cf1a2e5b145411b7c211efbeafe98b2ff5136fffdddd7"}
Nov 01 22:49:07.652866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.652816    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9" event={"ID":"caef116d-b8ec-4816-96a0-c69f81eb2f62","Type":"ContainerDied","Data":"9f52eb5b4ce838d44bfbdf510e070b0ea5a63773593587e9f3daa33069eb0c45"}
Nov 01 22:49:07.873745 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.873711    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:49:07.878033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:07.877993    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-k5fj9"]
Nov 01 22:49:08.173702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:08.173624    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" path="/var/lib/kubelet/pods/caef116d-b8ec-4816-96a0-c69f81eb2f62/volumes"
Nov 01 22:49:10.627045 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:49:10.627004    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:49:26.960566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:26.953528    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:49:26.980575 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:26.962619    5031 cpu_manager.go:414] "containerMap: removing container" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy"
Nov 01 22:49:26.980865 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:26.980834    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy"
Nov 01 22:49:26.981337 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:26.981310    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="caef116d-b8ec-4816-96a0-c69f81eb2f62" containerName="istio-proxy"
Nov 01 22:49:27.017883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.017846    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.027898 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.027861    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:49:27.132286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.132181    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.151867 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.151821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.152072 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.151894    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.152072 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.132658    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-68a7f708-serving-cert\""
Nov 01 22:49:27.152189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.152120    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfng\" (UniqueName: \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.253735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.253632    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.253735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.253698    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.254017 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.253852    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfng\" (UniqueName: \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.254104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.254068    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.254452 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.254420    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.255310 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:49:27.254571    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-serving-cert: secret "maas-api-e2e-derive-68a7f708-serving-cert" not found
Nov 01 22:49:27.255459 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:49:27.255371    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls podName:ba7b7890-6dbf-4835-b910-d59996c49ab0 nodeName:}" failed. No retries permitted until 2026-11-01 22:49:27.755348034 +0000 UTC m=+1521.711387856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls") pod "maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" (UID: "ba7b7890-6dbf-4835-b910-d59996c49ab0") : secret "maas-api-e2e-derive-68a7f708-serving-cert" not found
Nov 01 22:49:27.258408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.258378    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.324279 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.324238    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfng\" (UniqueName: \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.419320 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.419281    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 22:49:27.438096 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.438065    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.506736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.503913    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-a97162-openshift-default-dockercfg-m6mzd\""
Nov 01 22:49:27.507506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.507472    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 22:49:27.603070 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.603035    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.603407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.603381    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.603703 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.603681    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.603882 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.603866    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.604483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.604463    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.604647 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.604631    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.604778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.604762    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27xp\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.604930 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.604914    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.605048 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.605034    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.707565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.707504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.707915 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.707885    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.708104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.708087    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.708242 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.708226    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.708370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.708353    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27xp\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.708519 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.708504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.722799 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.722766    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.723139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.723112    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.723383 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.723365    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.730026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.714362    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.730260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.714878    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.730383 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.713591    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.730504 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.714170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.730689 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.713336    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.750638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.745117    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.750638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.748346    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.792617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.784723    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27xp\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.796402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.796055    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\") pod \"e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") " pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:27.835563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.831853    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.841578 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.840259    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") pod \"maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:27.946853 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:27.946819    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:28.062577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.061765    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:28.350655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.350516    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:49:28.368310 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:49:28.368275    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7b7890_6dbf_4835_b910_d59996c49ab0.slice/crio-db4e4b61f60646c421ff81a06db091d33c644a0110ca5b3eb635911faefa045e WatchSource:0}: Error finding container db4e4b61f60646c421ff81a06db091d33c644a0110ca5b3eb635911faefa045e: Status 404 returned error can't find the container with id db4e4b61f60646c421ff81a06db091d33c644a0110ca5b3eb635911faefa045e
Nov 01 22:49:28.500712 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.500676    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 22:49:28.504662 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:49:28.504620    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485be50_f54d_40c3_9644_08b2744e302c.slice/crio-e568e7e80b01c4594b382b0c2819084358399530d796695a58c6335f9fcfc54a WatchSource:0}: Error finding container e568e7e80b01c4594b382b0c2819084358399530d796695a58c6335f9fcfc54a: Status 404 returned error can't find the container with id e568e7e80b01c4594b382b0c2819084358399530d796695a58c6335f9fcfc54a
Nov 01 22:49:28.507719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.507681    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:49:28.507815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.507774    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:49:28.507815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.507804    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:49:28.833104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.833048    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerStarted","Data":"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"}
Nov 01 22:49:28.833329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.833118    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerStarted","Data":"e568e7e80b01c4594b382b0c2819084358399530d796695a58c6335f9fcfc54a"}
Nov 01 22:49:28.834838 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.834799    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" event={"ID":"ba7b7890-6dbf-4835-b910-d59996c49ab0","Type":"ContainerStarted","Data":"db4e4b61f60646c421ff81a06db091d33c644a0110ca5b3eb635911faefa045e"}
Nov 01 22:49:28.945846 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:28.945727    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podStartSLOduration=1.945706435 podStartE2EDuration="1.945706435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:28.944924279 +0000 UTC m=+1522.900964123" watchObservedRunningTime="2026-11-01 22:49:28.945706435 +0000 UTC m=+1522.901746280"
Nov 01 22:49:29.063054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.063007    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:29.074861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.074833    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:29.844511 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.844476    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" event={"ID":"ba7b7890-6dbf-4835-b910-d59996c49ab0","Type":"ContainerStarted","Data":"7c01b81f79348422a8d33e7e66be1f5cb005e31d2c219abdf72f7bab8b833f2d"}
Nov 01 22:49:29.844826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.844804    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:29.844889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.844852    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:29.845895 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.845874    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:49:29.962564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:29.962484    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" podStartSLOduration=3.062846135 podStartE2EDuration="3.962468786s" totalImagesPullingTime="899.622651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:29.959043879 +0000 UTC m=+1523.915083723" watchObservedRunningTime="2026-11-01 22:49:29.962468786 +0000 UTC m=+1523.918508630"
Nov 01 22:49:35.858411 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:49:35.858377    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:49:40.628127 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:49:40.628086    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:50:10.054052 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:10.054016    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"]
Nov 01 22:50:10.099267 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:10.099233    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ae82c8-jh68r"]
Nov 01 22:50:10.172293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:10.172257    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48438123-3ba2-44b4-90fd-f259922bdca4" path="/var/lib/kubelet/pods/48438123-3ba2-44b4-90fd-f259922bdca4/volumes"
Nov 01 22:50:10.626905 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:50:10.626866    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:50:28.858093 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.858058    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 22:50:28.862631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.862608    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:28.912375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.912348    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-a97162-serving-cert\""
Nov 01 22:50:28.929253 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.929214    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 22:50:28.995201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.995163    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:28.995402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.995216    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:28.995402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.995321    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5s9n\" (UniqueName: \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:28.995744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:28.995717    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.097747 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.097702    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.097949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.097850    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.098017 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.097973    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.098114 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:50:29.097895    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-serving-cert: secret "maas-api-e2e-shared-a-w0-a97162-serving-cert" not found
Nov 01 22:50:29.098213 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:50:29.098117    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls podName:26552a01-8249-404d-92cb-113439596f52 nodeName:}" failed. No retries permitted until 2026-11-01 22:50:29.598090479 +0000 UTC m=+1583.554130321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls") pod "maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" (UID: "26552a01-8249-404d-92cb-113439596f52") : secret "maas-api-e2e-shared-a-w0-a97162-serving-cert" not found
Nov 01 22:50:29.098346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.098021    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5s9n\" (UniqueName: \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.118113 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.098773    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.134327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.127583    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.154505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.154469    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5s9n\" (UniqueName: \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.612109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.612067    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.614893 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.614863    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:29.777138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:29.777102    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:30.158584 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:30.158523    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 22:50:30.163992 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:50:30.163962    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26552a01_8249_404d_92cb_113439596f52.slice/crio-137ffec9046522e5dcbc3de155afb548c4738708b03eb1d6a8103b6e21e602bc WatchSource:0}: Error finding container 137ffec9046522e5dcbc3de155afb548c4738708b03eb1d6a8103b6e21e602bc: Status 404 returned error can't find the container with id 137ffec9046522e5dcbc3de155afb548c4738708b03eb1d6a8103b6e21e602bc
Nov 01 22:50:30.165842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:30.165823    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:50:30.476003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:30.475903    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" event={"ID":"26552a01-8249-404d-92cb-113439596f52","Type":"ContainerStarted","Data":"137ffec9046522e5dcbc3de155afb548c4738708b03eb1d6a8103b6e21e602bc"}
Nov 01 22:50:31.484285 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:31.484185    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" event={"ID":"26552a01-8249-404d-92cb-113439596f52","Type":"ContainerStarted","Data":"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"}
Nov 01 22:50:31.484698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:31.484347    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:31.593699 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:31.593641    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" podStartSLOduration=2.676631418 podStartE2EDuration="3.593619425s" totalImagesPullingTime="916.988007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:31.592857944 +0000 UTC m=+1585.548897788" watchObservedRunningTime="2026-11-01 22:50:31.593619425 +0000 UTC m=+1585.549659270"
Nov 01 22:50:37.496773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:50:37.496743    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 22:50:40.626901 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:50:40.626856    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:51:07.510893 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:07.510863    5031 scope.go:122] "RemoveContainer" containerID="bbf4f9d5c003f8183b952092c5cca633cd90eb92cd15f74b950fe121c5fb36da"
Nov 01 22:51:10.628923 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:51:10.628884    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:51:11.850195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.850108    5031 generic.go:356] "Generic (PLEG): container finished" podID="f485be50-f54d-40c3-9644-08b2744e302c" containerID="41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6" exitCode=0
Nov 01 22:51:11.850748 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.850189    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerDied","Data":"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"}
Nov 01 22:51:11.850811 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.850796    5031 scope.go:122] "RemoveContainer" containerID="41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"
Nov 01 22:51:11.852625 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.852591    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:51:11.852728 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.852668    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:51:11.852728 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:11.852705    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:51:12.881294 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:12.881243    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerStarted","Data":"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"}
Nov 01 22:51:13.078783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:13.078717    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:51:13.080577 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:13.080520    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:13.080767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:13.080643    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:14.075565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:14.074720    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:14.075565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:14.074780    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:14.845033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:14.844997    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:51:15.068510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:15.068007    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:15.068510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:15.068090    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:16.074961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:16.074867    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:16.075938 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:16.075904    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:17.073563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:17.072009    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:17.073563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:17.072080    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:18.072044 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:18.072001    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 22:51:18.072687 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:18.072648    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 22:51:19.072749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:19.072716    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:51:19.075533 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:19.075506    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 22:51:24.797798 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:24.797754    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:51:24.798224 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:24.797986    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" podUID="ba7b7890-6dbf-4835-b910-d59996c49ab0" containerName="maas-api" containerID="cri-o://7c01b81f79348422a8d33e7e66be1f5cb005e31d2c219abdf72f7bab8b833f2d" gracePeriod=30
Nov 01 22:51:25.002783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.002747    5031 generic.go:356] "Generic (PLEG): container finished" podID="ba7b7890-6dbf-4835-b910-d59996c49ab0" containerID="7c01b81f79348422a8d33e7e66be1f5cb005e31d2c219abdf72f7bab8b833f2d" exitCode=0
Nov 01 22:51:25.003059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.002928    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" event={"ID":"ba7b7890-6dbf-4835-b910-d59996c49ab0","Type":"ContainerDied","Data":"7c01b81f79348422a8d33e7e66be1f5cb005e31d2c219abdf72f7bab8b833f2d"}
Nov 01 22:51:25.134084 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.134054    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:51:25.245532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.245489    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") pod \"ba7b7890-6dbf-4835-b910-d59996c49ab0\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") "
Nov 01 22:51:25.245752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.245598    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\" (UniqueName: \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\") pod \"ba7b7890-6dbf-4835-b910-d59996c49ab0\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") "
Nov 01 22:51:25.245752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.245621    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\") pod \"ba7b7890-6dbf-4835-b910-d59996c49ab0\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") "
Nov 01 22:51:25.245884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.245789    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\") pod \"ba7b7890-6dbf-4835-b910-d59996c49ab0\" (UID: \"ba7b7890-6dbf-4835-b910-d59996c49ab0\") "
Nov 01 22:51:25.246169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.246127    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca" pod "ba7b7890-6dbf-4835-b910-d59996c49ab0" (UID: "ba7b7890-6dbf-4835-b910-d59996c49ab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:25.246680 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.246661    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7b7890-6dbf-4835-b910-d59996c49ab0-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:25.248609 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.248527    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs" pod "ba7b7890-6dbf-4835-b910-d59996c49ab0" (UID: "ba7b7890-6dbf-4835-b910-d59996c49ab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:51:25.249021 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.248995    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng" pod "ba7b7890-6dbf-4835-b910-d59996c49ab0" (UID: "ba7b7890-6dbf-4835-b910-d59996c49ab0"). InnerVolumeSpecName "kube-api-access-fqfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:25.250901 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.250874    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls" pod "ba7b7890-6dbf-4835-b910-d59996c49ab0" (UID: "ba7b7890-6dbf-4835-b910-d59996c49ab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:51:25.349710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.349666    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:25.349710 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.349703    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7b7890-6dbf-4835-b910-d59996c49ab0-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:25.349992 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:25.349720    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfng\" (UniqueName: \"kubernetes.io/projected/ba7b7890-6dbf-4835-b910-d59996c49ab0-kube-api-access-fqfng\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:26.037558 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:26.037437    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl" event={"ID":"ba7b7890-6dbf-4835-b910-d59996c49ab0","Type":"ContainerDied","Data":"db4e4b61f60646c421ff81a06db091d33c644a0110ca5b3eb635911faefa045e"}
Nov 01 22:51:26.037558 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:26.037522    5031 scope.go:122] "RemoveContainer" containerID="7c01b81f79348422a8d33e7e66be1f5cb005e31d2c219abdf72f7bab8b833f2d"
Nov 01 22:51:26.038116 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:26.037712    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"
Nov 01 22:51:26.196323 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:26.196286    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:51:26.241906 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:26.241874    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-68a7f708-7bc7f86b97-wltnl"]
Nov 01 22:51:28.174981 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:28.174944    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7b7890-6dbf-4835-b910-d59996c49ab0" path="/var/lib/kubelet/pods/ba7b7890-6dbf-4835-b910-d59996c49ab0/volumes"
Nov 01 22:51:29.763437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.763398    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:51:29.763996 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.763667    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy" containerID="cri-o://89aaf269fc703f1a44096f9c1558871dd5e8f98a1520bec617127214a36d88d7" gracePeriod=30
Nov 01 22:51:29.809561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.809511    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:51:29.809775 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.809742    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" containerID="cri-o://7aba0af6bd37810833fb0916113b2f27aacab2e00cf176fe55bf06b09ffdde59" gracePeriod=30
Nov 01 22:51:29.856442 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.856406    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:51:29.856988 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.856952    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy" containerID="cri-o://eaf93156c3a895dc34d8bdaac0cfe40b6066df4948acedbe402c5476271682b8" gracePeriod=30
Nov 01 22:51:29.866557 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.866512    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:51:29.866832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.866807    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" containerID="cri-o://3d729305073c40904a5a9cb87c5b7d53fa815eb95a3c5316362c8b77f80fd849" gracePeriod=30
Nov 01 22:51:29.871453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.871422    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:51:29.871677 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:29.871656    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" containerID="cri-o://c45edfdbc8ebb8583362c7439a9a03d021993cd3f3a0716a5876af90cd3b58fa" gracePeriod=30
Nov 01 22:51:33.203244 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:33.203202    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:33.203732 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:33.203268    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:33.480722 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:33.479646    5031 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:33.480722 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:33.479742    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:35.112596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.110061    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 22:51:35.112596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.110127    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 22:51:35.130560 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.128771    5031 generic.go:356] "Generic (PLEG): container finished" podID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerID="7aba0af6bd37810833fb0916113b2f27aacab2e00cf176fe55bf06b09ffdde59" exitCode=0
Nov 01 22:51:35.130560 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.128942    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" event={"ID":"b90d7a16-3248-4931-bb95-9f87511cbc17","Type":"ContainerDied","Data":"7aba0af6bd37810833fb0916113b2f27aacab2e00cf176fe55bf06b09ffdde59"}
Nov 01 22:51:35.136243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.132267    5031 generic.go:356] "Generic (PLEG): container finished" podID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerID="eaf93156c3a895dc34d8bdaac0cfe40b6066df4948acedbe402c5476271682b8" exitCode=0
Nov 01 22:51:35.136243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.132405    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" event={"ID":"ae11ef97-fb35-4928-9e06-091d73907b5f","Type":"ContainerDied","Data":"eaf93156c3a895dc34d8bdaac0cfe40b6066df4948acedbe402c5476271682b8"}
Nov 01 22:51:35.136243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.134353    5031 generic.go:356] "Generic (PLEG): container finished" podID="499797d4-f4ee-4657-b153-62e754ac928f" containerID="89aaf269fc703f1a44096f9c1558871dd5e8f98a1520bec617127214a36d88d7" exitCode=0
Nov 01 22:51:35.136243 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.134515    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" event={"ID":"499797d4-f4ee-4657-b153-62e754ac928f","Type":"ContainerDied","Data":"89aaf269fc703f1a44096f9c1558871dd5e8f98a1520bec617127214a36d88d7"}
Nov 01 22:51:35.159406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.159364    5031 generic.go:356] "Generic (PLEG): container finished" podID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerID="c45edfdbc8ebb8583362c7439a9a03d021993cd3f3a0716a5876af90cd3b58fa" exitCode=0
Nov 01 22:51:35.159665 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.159402    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" event={"ID":"bf6212b9-7ec6-4561-9017-5286c3a0efb4","Type":"ContainerDied","Data":"c45edfdbc8ebb8583362c7439a9a03d021993cd3f3a0716a5876af90cd3b58fa"}
Nov 01 22:51:35.173737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.172806    5031 generic.go:356] "Generic (PLEG): container finished" podID="c90369e1-46be-40a6-825a-c30caf3481a6" containerID="3d729305073c40904a5a9cb87c5b7d53fa815eb95a3c5316362c8b77f80fd849" exitCode=0
Nov 01 22:51:35.173737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.172860    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" event={"ID":"c90369e1-46be-40a6-825a-c30caf3481a6","Type":"ContainerDied","Data":"3d729305073c40904a5a9cb87c5b7d53fa815eb95a3c5316362c8b77f80fd849"}
Nov 01 22:51:35.184327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.183970    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:51:35.224397 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.224359    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.224641 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.224442    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.224641 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.224602    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.224832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.224768    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.225030 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.224991    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:35.225329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225304    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.225750 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225369    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.225931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225864    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.225931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225900    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.226055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225974    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.226055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.226041    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\") pod \"499797d4-f4ee-4657-b153-62e754ac928f\" (UID: \"499797d4-f4ee-4657-b153-62e754ac928f\") "
Nov 01 22:51:35.227041 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.227018    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.227138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.227047    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499797d4-f4ee-4657-b153-62e754ac928f-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.227686 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.225719    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.227783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.227322    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.227783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.227400    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "kube-api-access-xfskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.227783 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.227635    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.229207 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.229178    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.229909 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.229887    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.230009 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.229989    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo" pod "499797d4-f4ee-4657-b153-62e754ac928f" (UID: "499797d4-f4ee-4657-b153-62e754ac928f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371738    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371781    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371806    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499797d4-f4ee-4657-b153-62e754ac928f-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371949    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371964    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371979    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499797d4-f4ee-4657-b153-62e754ac928f-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.372842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.371999    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfskg\" (UniqueName: \"kubernetes.io/projected/499797d4-f4ee-4657-b153-62e754ac928f-kube-api-access-xfskg\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.443662 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.439893    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:51:35.568776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.559400    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:51:35.574232 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.574195    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:51:35.585128 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585090    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585212    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585261    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585350 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585306    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585380    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585474 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585518 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585504    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585636 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585621    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.585820 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.585806    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\") pod \"ae11ef97-fb35-4928-9e06-091d73907b5f\" (UID: \"ae11ef97-fb35-4928-9e06-091d73907b5f\") "
Nov 01 22:51:35.586892 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.586867    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.587116 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.587098    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.587277 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.587261    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.588136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.588110    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.588679 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.588657    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:35.590164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.590121    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.592366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.592282    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "kube-api-access-9m8bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.594581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.593935    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.595942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.595914    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo" pod "ae11ef97-fb35-4928-9e06-091d73907b5f" (UID: "ae11ef97-fb35-4928-9e06-091d73907b5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:35.596106 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.596086    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:51:35.688334 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688230    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.688532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688339    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.688532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688380    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.688532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688429    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.688532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688499    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.688793 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.688527    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689163 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689044    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689163 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689147    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.689334 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689197    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689334 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689260    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.689445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689355    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689372    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.689445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689402    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689436    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689464    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689522    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689588    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.689685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689620    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689647    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.689946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689706    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689752    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.689946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689782    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.689946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689814    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.689946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689907    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.690201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.689987    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.690201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.690021    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\") pod \"c90369e1-46be-40a6-825a-c30caf3481a6\" (UID: \"c90369e1-46be-40a6-825a-c30caf3481a6\") "
Nov 01 22:51:35.690201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.690086    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\") pod \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\" (UID: \"bf6212b9-7ec6-4561-9017-5286c3a0efb4\") "
Nov 01 22:51:35.690201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.690145    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\") pod \"b90d7a16-3248-4931-bb95-9f87511cbc17\" (UID: \"b90d7a16-3248-4931-bb95-9f87511cbc17\") "
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691826    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691858    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8bb\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-kube-api-access-9m8bb\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691873    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691887    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae11ef97-fb35-4928-9e06-091d73907b5f-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691900    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691914    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691928    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691942    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691957    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae11ef97-fb35-4928-9e06-091d73907b5f-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.692157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.691971    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae11ef97-fb35-4928-9e06-091d73907b5f-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.696382 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.696340    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.696650    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.696710    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "kube-api-access-km6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.698630    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.699158    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.699316    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.699472    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.699500    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.699946    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.700190    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.700384    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.700686    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.700975    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.701385    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.702136 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.701785    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.703148    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.703262    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.703355    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.705361    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.705518    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "kube-api-access-w6tl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.705651    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.705691    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.706200    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token" pod "c90369e1-46be-40a6-825a-c30caf3481a6" (UID: "c90369e1-46be-40a6-825a-c30caf3481a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.706463    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "kube-api-access-fxn62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:35.706735 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.706517    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy" pod "b90d7a16-3248-4931-bb95-9f87511cbc17" (UID: "b90d7a16-3248-4931-bb95-9f87511cbc17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:35.707566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.707526    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo" pod "bf6212b9-7ec6-4561-9017-5286c3a0efb4" (UID: "bf6212b9-7ec6-4561-9017-5286c3a0efb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794573    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794622    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794641    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90d7a16-3248-4931-bb95-9f87511cbc17-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794655    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794670    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tl2\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-kube-api-access-w6tl2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794696    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c90369e1-46be-40a6-825a-c30caf3481a6-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794718    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794739    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.794759    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795631    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795650    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795664    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxn62\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-kube-api-access-fxn62\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795679    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795692    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795706    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795719    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795731    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795745    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795758    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6pk\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-kube-api-access-km6pk\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795772    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795786    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c90369e1-46be-40a6-825a-c30caf3481a6-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795800    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90d7a16-3248-4931-bb95-9f87511cbc17-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795812    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf6212b9-7ec6-4561-9017-5286c3a0efb4-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795824    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c90369e1-46be-40a6-825a-c30caf3481a6-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795836    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90d7a16-3248-4931-bb95-9f87511cbc17-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:35.806574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:35.795849    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c90369e1-46be-40a6-825a-c30caf3481a6-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:36.184164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.183975    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"
Nov 01 22:51:36.184164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.183990    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g" event={"ID":"ae11ef97-fb35-4928-9e06-091d73907b5f","Type":"ContainerDied","Data":"46d0026893d186b9fac3fe114f70b4e8842b9ca7c6a60952f3a9630d1d76d4bb"}
Nov 01 22:51:36.184164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.184057    5031 scope.go:122] "RemoveContainer" containerID="eaf93156c3a895dc34d8bdaac0cfe40b6066df4948acedbe402c5476271682b8"
Nov 01 22:51:36.186137 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.186115    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"
Nov 01 22:51:36.186245 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.186167    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c" event={"ID":"499797d4-f4ee-4657-b153-62e754ac928f","Type":"ContainerDied","Data":"b489a42310331d56b607042bcc089befea4545cf28cf4138d982ca3ef4772a51"}
Nov 01 22:51:36.188187 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.187948    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"
Nov 01 22:51:36.188187 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.187988    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj" event={"ID":"bf6212b9-7ec6-4561-9017-5286c3a0efb4","Type":"ContainerDied","Data":"884139e916e023e2cd62f938a5ddba59658ddac86833f741a7ef9a89fac79a2d"}
Nov 01 22:51:36.190441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.190415    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"
Nov 01 22:51:36.190589 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.190433    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj" event={"ID":"c90369e1-46be-40a6-825a-c30caf3481a6","Type":"ContainerDied","Data":"1453b89d7f4488ab71c148d5fcad8ae032a9eee97480a8fca7365c669d33487f"}
Nov 01 22:51:36.192143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.192116    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw" event={"ID":"b90d7a16-3248-4931-bb95-9f87511cbc17","Type":"ContainerDied","Data":"8087e3dadc6ac2d9ba248a3870a08bf25b4796ba52c25938ac0d4eb034d1eb53"}
Nov 01 22:51:36.192143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.192130    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"
Nov 01 22:51:36.206740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.206718    5031 scope.go:122] "RemoveContainer" containerID="89aaf269fc703f1a44096f9c1558871dd5e8f98a1520bec617127214a36d88d7"
Nov 01 22:51:36.216862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.216842    5031 scope.go:122] "RemoveContainer" containerID="c45edfdbc8ebb8583362c7439a9a03d021993cd3f3a0716a5876af90cd3b58fa"
Nov 01 22:51:36.227963 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.227937    5031 scope.go:122] "RemoveContainer" containerID="3d729305073c40904a5a9cb87c5b7d53fa815eb95a3c5316362c8b77f80fd849"
Nov 01 22:51:36.238089 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.238069    5031 scope.go:122] "RemoveContainer" containerID="7aba0af6bd37810833fb0916113b2f27aacab2e00cf176fe55bf06b09ffdde59"
Nov 01 22:51:36.340187 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.340153    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:51:36.385330 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.385297    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-4bz6c"]
Nov 01 22:51:36.449867 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.449784    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:51:36.455595 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.455565    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zlcnj"]
Nov 01 22:51:36.560004 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.559965    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:51:36.566460 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.566428    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-zzslj"]
Nov 01 22:51:36.668155 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.668118    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:51:36.675058 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.674458    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-wz68g"]
Nov 01 22:51:36.782313 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.782213    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:51:36.787478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:36.787446    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-jkszw"]
Nov 01 22:51:38.173038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:38.173004    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499797d4-f4ee-4657-b153-62e754ac928f" path="/var/lib/kubelet/pods/499797d4-f4ee-4657-b153-62e754ac928f/volumes"
Nov 01 22:51:38.173703 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:38.173687    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" path="/var/lib/kubelet/pods/ae11ef97-fb35-4928-9e06-091d73907b5f/volumes"
Nov 01 22:51:38.174168 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:38.174153    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" path="/var/lib/kubelet/pods/b90d7a16-3248-4931-bb95-9f87511cbc17/volumes"
Nov 01 22:51:38.174629 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:38.174616    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" path="/var/lib/kubelet/pods/bf6212b9-7ec6-4561-9017-5286c3a0efb4/volumes"
Nov 01 22:51:38.175349 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:38.175332    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" path="/var/lib/kubelet/pods/c90369e1-46be-40a6-825a-c30caf3481a6/volumes"
Nov 01 22:51:40.627633 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:51:40.627589    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:51:41.648821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.648781    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 22:51:41.649660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649638    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ba7b7890-6dbf-4835-b910-d59996c49ab0" containerName="maas-api"
Nov 01 22:51:41.649660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649662    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7b7890-6dbf-4835-b910-d59996c49ab0" containerName="maas-api"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649682    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649689    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649713    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649720    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649736    5031 cpu_manager.go:414] "containerMap: removing container" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649743    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649755    5031 cpu_manager.go:414] "containerMap: removing container" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649762    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649778    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy"
Nov 01 22:51:41.649830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649786    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649960    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="499797d4-f4ee-4657-b153-62e754ac928f" containerName="istio-proxy"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649977    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6212b9-7ec6-4561-9017-5286c3a0efb4" containerName="istio-proxy"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.649991    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7b7890-6dbf-4835-b910-d59996c49ab0" containerName="maas-api"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.650002    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90369e1-46be-40a6-825a-c30caf3481a6" containerName="istio-proxy"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.650012    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae11ef97-fb35-4928-9e06-091d73907b5f" containerName="istio-proxy"
Nov 01 22:51:41.650289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.650024    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90d7a16-3248-4931-bb95-9f87511cbc17" containerName="istio-proxy"
Nov 01 22:51:41.655871 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.655843    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.674228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.674193    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvm64\" (UniqueName: \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.674395 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.674256    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.704205 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.704172    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8bac2f\""
Nov 01 22:51:41.759130 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.759085    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8bac2f-dockercfg-xr92w\""
Nov 01 22:51:41.764016 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.763977    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 22:51:41.780660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.780619    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvm64\" (UniqueName: \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.780861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.780702    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.781757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.781731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.807043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.807007    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 22:51:41.817522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.817489    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:41.835973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.835940    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvm64\" (UniqueName: \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\") pod \"payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.862574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.862519    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 22:51:41.882972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.882932    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck84v\" (UniqueName: \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:41.883194 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.883168    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:41.974616 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.971199    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:41.988412 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.988351    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck84v\" (UniqueName: \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:41.988654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.988560    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:41.989640 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:41.989593    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:42.043412 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:42.043384    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck84v\" (UniqueName: \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\") pod \"payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:42.140397 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:42.140101    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:42.396018 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:42.395985    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 22:51:42.400066 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:51:42.400034    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0ff0e9_cf70_42a2_a30b_d0d26adb3ba8.slice/crio-cca74b53251bccd0b4cfb3eb8ff359385873ec2b72aecaf1316c0961099fd295 WatchSource:0}: Error finding container cca74b53251bccd0b4cfb3eb8ff359385873ec2b72aecaf1316c0961099fd295: Status 404 returned error can't find the container with id cca74b53251bccd0b4cfb3eb8ff359385873ec2b72aecaf1316c0961099fd295
Nov 01 22:51:42.583254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:42.583218    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 22:51:42.587485 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:51:42.587458    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45d2f1a_36c0_4f86_bd72_50010f398ac7.slice/crio-9eff532b9ebd409467b7dc3d9dfee1cf489a524c7ee40d1a2330c5bd94e443c9 WatchSource:0}: Error finding container 9eff532b9ebd409467b7dc3d9dfee1cf489a524c7ee40d1a2330c5bd94e443c9: Status 404 returned error can't find the container with id 9eff532b9ebd409467b7dc3d9dfee1cf489a524c7ee40d1a2330c5bd94e443c9
Nov 01 22:51:43.291075 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:43.290979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" event={"ID":"d45d2f1a-36c0-4f86-bd72-50010f398ac7","Type":"ContainerStarted","Data":"9eff532b9ebd409467b7dc3d9dfee1cf489a524c7ee40d1a2330c5bd94e443c9"}
Nov 01 22:51:43.292316 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:43.292293    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" event={"ID":"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8","Type":"ContainerStarted","Data":"cca74b53251bccd0b4cfb3eb8ff359385873ec2b72aecaf1316c0961099fd295"}
Nov 01 22:51:46.322598 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.322562    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" event={"ID":"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8","Type":"ContainerStarted","Data":"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"}
Nov 01 22:51:46.323056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.322648    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:46.324298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.324273    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" event={"ID":"d45d2f1a-36c0-4f86-bd72-50010f398ac7","Type":"ContainerStarted","Data":"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"}
Nov 01 22:51:46.324436 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.324414    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:51:46.433698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.433642    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" podStartSLOduration=2.267167211 podStartE2EDuration="5.433624753s" totalImagesPullingTime="3.166457542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:46.431670684 +0000 UTC m=+1660.387710528" watchObservedRunningTime="2026-11-01 22:51:46.433624753 +0000 UTC m=+1660.389664596"
Nov 01 22:51:46.541183 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:46.541126    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" podStartSLOduration=2.271426198 podStartE2EDuration="5.541110362s" totalImagesPullingTime="3.269684164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:46.538796698 +0000 UTC m=+1660.494836544" watchObservedRunningTime="2026-11-01 22:51:46.541110362 +0000 UTC m=+1660.497150205"
Nov 01 22:51:49.215321 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.215284    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 22:51:49.220318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.220288    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.268653 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.268623    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-2b0866-dockercfg-4thch\""
Nov 01 22:51:49.327864 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.323554    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-2b0866\""
Nov 01 22:51:49.329461 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.329425    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 22:51:49.410205 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.410168    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg4c\" (UniqueName: \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.410741 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.410714    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.454391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.420818    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 22:51:49.477613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.473871    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 22:51:49.477613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.473992    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.512841 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.512788    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg4c\" (UniqueName: \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.513051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.513008    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.513919 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.513887    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.566834 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.566793    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg4c\" (UniqueName: \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\") pod \"payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:49.614132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.614092    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbstw\" (UniqueName: \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.614326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.614228    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.716030 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.715992    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbstw\" (UniqueName: \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.716281 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.716107    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.716874 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.716853    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.769469 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.769387    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbstw\" (UniqueName: \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\") pod \"payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.798830 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.798796    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:49.834815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:49.834778    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:50.186238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.186204    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 22:51:50.190299 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:51:50.190265    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5888fbd6_9378_4282_ad03_6c472c5b67c5.slice/crio-426d5162a632ff719cea6c7cdaea6ef1fac68871464cfc1b6d9437545dc5a33b WatchSource:0}: Error finding container 426d5162a632ff719cea6c7cdaea6ef1fac68871464cfc1b6d9437545dc5a33b: Status 404 returned error can't find the container with id 426d5162a632ff719cea6c7cdaea6ef1fac68871464cfc1b6d9437545dc5a33b
Nov 01 22:51:50.235583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.235474    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 22:51:50.239772 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:51:50.239733    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01eba096_753a_48c3_8f7f_6ec6f1a99e35.slice/crio-f8d47745d7a7029937c2d380adb11c6985b08e4082387119bb895dcd52e47bed WatchSource:0}: Error finding container f8d47745d7a7029937c2d380adb11c6985b08e4082387119bb895dcd52e47bed: Status 404 returned error can't find the container with id f8d47745d7a7029937c2d380adb11c6985b08e4082387119bb895dcd52e47bed
Nov 01 22:51:50.485983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.485891    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" event={"ID":"01eba096-753a-48c3-8f7f-6ec6f1a99e35","Type":"ContainerStarted","Data":"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"}
Nov 01 22:51:50.485983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.485950    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" event={"ID":"01eba096-753a-48c3-8f7f-6ec6f1a99e35","Type":"ContainerStarted","Data":"f8d47745d7a7029937c2d380adb11c6985b08e4082387119bb895dcd52e47bed"}
Nov 01 22:51:50.485983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.485980    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:51:50.487941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.487910    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" event={"ID":"5888fbd6-9378-4282-ad03-6c472c5b67c5","Type":"ContainerStarted","Data":"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"}
Nov 01 22:51:50.488085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.487973    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:51:50.488085 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.487993    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" event={"ID":"5888fbd6-9378-4282-ad03-6c472c5b67c5","Type":"ContainerStarted","Data":"426d5162a632ff719cea6c7cdaea6ef1fac68871464cfc1b6d9437545dc5a33b"}
Nov 01 22:51:50.594400 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.594349    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" podStartSLOduration=1.594336134 podStartE2EDuration="1.594336134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:50.592015734 +0000 UTC m=+1664.548055578" watchObservedRunningTime="2026-11-01 22:51:50.594336134 +0000 UTC m=+1664.550375978"
Nov 01 22:51:50.700931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:50.700870    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" podStartSLOduration=1.7008543280000001 podStartE2EDuration="1.700854328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:50.697823916 +0000 UTC m=+1664.653863760" watchObservedRunningTime="2026-11-01 22:51:50.700854328 +0000 UTC m=+1664.656894200"
Nov 01 22:51:57.336572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:57.336510    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 22:51:57.337096 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:51:57.336618    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 22:52:01.497434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:01.497377    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 22:52:01.497939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:01.497469    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 22:52:10.630765 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:10.630726    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:52:14.950718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:14.950679    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:52:14.951264 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:14.950945    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy" containerID="cri-o://1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72" gracePeriod=30
Nov 01 22:52:14.997217 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:14.997176    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:52:14.997455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:14.997427    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy" containerID="cri-o://d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454" gracePeriod=30
Nov 01 22:52:15.000819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.000785    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:52:15.001078 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.001047    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" containerID="cri-o://68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3" gracePeriod=30
Nov 01 22:52:15.043098 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.043032    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:52:15.043455 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.043306    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy" containerID="cri-o://10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d" gracePeriod=30
Nov 01 22:52:15.105859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.103520    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:52:15.105859 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.103840    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy" containerID="cri-o://9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04" gracePeriod=30
Nov 01 22:52:15.107974 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.107943    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:52:15.108200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.108174    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy" containerID="cri-o://5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5" gracePeriod=30
Nov 01 22:52:15.111478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.111455    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:52:15.111792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.111739    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" containerID="cri-o://653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2" gracePeriod=30
Nov 01 22:52:15.115707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.115671    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:52:15.115979 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:15.115952    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy" containerID="cri-o://fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9" gracePeriod=30
Nov 01 22:52:17.390942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:17.390907    5031 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:52:17.391328 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:17.390974    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:52:17.605213 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:17.605177    5031 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:52:17.605402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:17.605234    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:52:20.447145 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.441180    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:52:20.454762 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.454730    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:52:20.513083 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513047    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513195    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513228    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513303    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513380 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513364    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513419 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513405    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.513472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513457    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513670    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513712    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\") pod \"fb4b1daa-bf3e-403d-845b-7558ca6165be\" (UID: \"fb4b1daa-bf3e-403d-845b-7558ca6165be\") "
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.513890    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514072    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514279    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514356    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514415    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514502    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514524    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb4b1daa-bf3e-403d-845b-7558ca6165be-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514554    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.514707 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.514567    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.516945 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.516904    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.517373 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.517336    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.519624 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.519588    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "kube-api-access-62h46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.525763 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.525728    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token" pod "fb4b1daa-bf3e-403d-845b-7558ca6165be" (UID: "fb4b1daa-bf3e-403d-845b-7558ca6165be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.563351 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.563237    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:52:20.632720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.632628    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.632905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.632780    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.632905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.632882    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.632924    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.632997    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633192 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.633047    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633192 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.633084    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633304 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.633207    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.633304 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.633248    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\") pod \"185b08a2-1ec2-418f-90b1-43b584d46163\" (UID: \"185b08a2-1ec2-418f-90b1-43b584d46163\") "
Nov 01 22:52:20.634401 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634370    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.634401 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634403    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.634583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634416    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb4b1daa-bf3e-403d-845b-7558ca6165be-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.634583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634429    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62h46\" (UniqueName: \"kubernetes.io/projected/fb4b1daa-bf3e-403d-845b-7558ca6165be-kube-api-access-62h46\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.634583 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634443    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb4b1daa-bf3e-403d-845b-7558ca6165be-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.634734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.634675    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.635593 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.635105    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.635593 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.635268    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.638592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.636300    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.638592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.638457    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "kube-api-access-24jpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.639850 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.636822    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.639973 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.639942    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.642069 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.642029    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.643920 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.643888    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy" pod "185b08a2-1ec2-418f-90b1-43b584d46163" (UID: "185b08a2-1ec2-418f-90b1-43b584d46163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.649101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.649075    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:52:20.657627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.657595    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:52:20.664467 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.664435    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:52:20.669447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.669413    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:52:20.679157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.679124    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:52:20.701113 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.701076    5031 generic.go:356] "Generic (PLEG): container finished" podID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerID="9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04" exitCode=0
Nov 01 22:52:20.701291 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.701222    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" event={"ID":"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12","Type":"ContainerDied","Data":"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"}
Nov 01 22:52:20.701291 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.701286    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h" event={"ID":"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12","Type":"ContainerDied","Data":"5d1448bcf7b4829d560787cb7c161d659bc1ac9574f7b5a9e5d2aaa030d60378"}
Nov 01 22:52:20.701370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.701306    5031 scope.go:122] "RemoveContainer" containerID="9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"
Nov 01 22:52:20.701419 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.701408    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"
Nov 01 22:52:20.703364 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.703333    5031 generic.go:356] "Generic (PLEG): container finished" podID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerID="10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d" exitCode=0
Nov 01 22:52:20.703480 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.703403    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"
Nov 01 22:52:20.703480 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.703414    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" event={"ID":"d9ddc310-ff4a-407e-b55b-304ffb3fd629","Type":"ContainerDied","Data":"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"}
Nov 01 22:52:20.703480 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.703464    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs" event={"ID":"d9ddc310-ff4a-407e-b55b-304ffb3fd629","Type":"ContainerDied","Data":"fa567b02075bce128ce1fbfdb2431ec47d42bc8bf69ba41feea699335b301160"}
Nov 01 22:52:20.705226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.705196    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerID="653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2" exitCode=0
Nov 01 22:52:20.705365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.705257    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"
Nov 01 22:52:20.705365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.705295    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" event={"ID":"7d484274-310f-4149-be9d-6a62c1f8ae93","Type":"ContainerDied","Data":"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"}
Nov 01 22:52:20.705365 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.705339    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf" event={"ID":"7d484274-310f-4149-be9d-6a62c1f8ae93","Type":"ContainerDied","Data":"2aaec39896a96b85da7026f0e612e9f22d21958d34c63bebb98d5fb45c2287e2"}
Nov 01 22:52:20.706847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.706822    5031 generic.go:356] "Generic (PLEG): container finished" podID="6db83d80-352b-4a54-93e1-427d5aadff08" containerID="fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9" exitCode=0
Nov 01 22:52:20.706958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.706878    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"
Nov 01 22:52:20.706958 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.706903    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" event={"ID":"6db83d80-352b-4a54-93e1-427d5aadff08","Type":"ContainerDied","Data":"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"}
Nov 01 22:52:20.707353 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.707247    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf" event={"ID":"6db83d80-352b-4a54-93e1-427d5aadff08","Type":"ContainerDied","Data":"bf296c5945827f330cdd61472bb877497754a48e1d3b5991363923fe276d2f60"}
Nov 01 22:52:20.708608 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.708581    5031 generic.go:356] "Generic (PLEG): container finished" podID="185b08a2-1ec2-418f-90b1-43b584d46163" containerID="d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454" exitCode=0
Nov 01 22:52:20.708706 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.708692    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" event={"ID":"185b08a2-1ec2-418f-90b1-43b584d46163","Type":"ContainerDied","Data":"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"}
Nov 01 22:52:20.708753 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.708733    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8" event={"ID":"185b08a2-1ec2-418f-90b1-43b584d46163","Type":"ContainerDied","Data":"d3f69ac03ea23178daaf3e22263a26b1c7dd3e2f48313b0e085ba118d1888fb1"}
Nov 01 22:52:20.708814 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.708801    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"
Nov 01 22:52:20.711120 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.711093    5031 generic.go:356] "Generic (PLEG): container finished" podID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerID="1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72" exitCode=0
Nov 01 22:52:20.711276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.711179    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" event={"ID":"fb4b1daa-bf3e-403d-845b-7558ca6165be","Type":"ContainerDied","Data":"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"}
Nov 01 22:52:20.711276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.711217    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp" event={"ID":"fb4b1daa-bf3e-403d-845b-7558ca6165be","Type":"ContainerDied","Data":"076caf81e5935a6887152b4f83b68d5dd988f89786e2316b31ca7c0763365039"}
Nov 01 22:52:20.711456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.711296    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"
Nov 01 22:52:20.714714 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.714692    5031 scope.go:122] "RemoveContainer" containerID="9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"
Nov 01 22:52:20.715286 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.715253    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04\": container with ID starting with 9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04 not found: ID does not exist" containerID="9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"
Nov 01 22:52:20.715381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.715299    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04"} err="failed to get container status \"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04\": rpc error: code = NotFound desc = could not find container \"9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04\": container with ID starting with 9ab744e84823678a2fdabd312900c463df15f36c6be47afbf0157a0c7d385e04 not found: ID does not exist"
Nov 01 22:52:20.715381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.715324    5031 scope.go:122] "RemoveContainer" containerID="10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"
Nov 01 22:52:20.718169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.717853    5031 generic.go:356] "Generic (PLEG): container finished" podID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerID="68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3" exitCode=0
Nov 01 22:52:20.718169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.717966    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" event={"ID":"6dcabe3e-1f3b-4471-904d-fd934a18a547","Type":"ContainerDied","Data":"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"}
Nov 01 22:52:20.718169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.718015    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c" event={"ID":"6dcabe3e-1f3b-4471-904d-fd934a18a547","Type":"ContainerDied","Data":"9e14e025ea809e61a728ff8f22550848e3a52b55660568dbcf981e82ffa39860"}
Nov 01 22:52:20.718169 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.718091    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"
Nov 01 22:52:20.721300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.721272    5031 generic.go:356] "Generic (PLEG): container finished" podID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerID="5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5" exitCode=0
Nov 01 22:52:20.721402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.721317    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerDied","Data":"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"}
Nov 01 22:52:20.721402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.721332    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"
Nov 01 22:52:20.721402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.721362    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf" event={"ID":"ae3a252d-49cf-415c-8a86-80e048220cd4","Type":"ContainerDied","Data":"4a7235e21ef9d00a24021ca5fe6095f0ccfb79c992797c02fdb491ad7f294f6f"}
Nov 01 22:52:20.729199 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.729180    5031 scope.go:122] "RemoveContainer" containerID="10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"
Nov 01 22:52:20.729554 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.729519    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d\": container with ID starting with 10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d not found: ID does not exist" containerID="10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"
Nov 01 22:52:20.729638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.729598    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d"} err="failed to get container status \"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d\": rpc error: code = NotFound desc = could not find container \"10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d\": container with ID starting with 10555eb1cc7a5ba88430af012062354369c5c37cfa82cd26ccef6530c2abe21d not found: ID does not exist"
Nov 01 22:52:20.729638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.729625    5031 scope.go:122] "RemoveContainer" containerID="653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"
Nov 01 22:52:20.735153 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735080    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735153 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735138    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735225    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735372 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735299    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735509 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735492    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735559    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.735658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735593    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735658 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735636    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735825 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735679    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.735825 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735695    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735825 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\") pod \"6dcabe3e-1f3b-4471-904d-fd934a18a547\" (UID: \"6dcabe3e-1f3b-4471-904d-fd934a18a547\") "
Nov 01 22:52:20.735972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.735822    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.736141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736119    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736317    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736525    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736567    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736583    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736597    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736610    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736624    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jpb\" (UniqueName: \"kubernetes.io/projected/185b08a2-1ec2-418f-90b1-43b584d46163-kube-api-access-24jpb\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736639    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b08a2-1ec2-418f-90b1-43b584d46163-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736653    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736668    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736681    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736694    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcabe3e-1f3b-4471-904d-fd934a18a547-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736707    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b08a2-1ec2-418f-90b1-43b584d46163-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736722    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.736734    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b08a2-1ec2-418f-90b1-43b584d46163-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.738581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.738212    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "kube-api-access-f42w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.739443 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.738604    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.739443 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.739196    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.742258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.742232    5031 scope.go:122] "RemoveContainer" containerID="653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"
Nov 01 22:52:20.742662 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.742630    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2\": container with ID starting with 653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2 not found: ID does not exist" containerID="653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"
Nov 01 22:52:20.742772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.742672    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2"} err="failed to get container status \"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2\": rpc error: code = NotFound desc = could not find container \"653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2\": container with ID starting with 653fc3ca66e496f7c06f2c13dd4b2e40f6dedc3c4b424c5b10ce185d5ae9b5b2 not found: ID does not exist"
Nov 01 22:52:20.742772 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.742693    5031 scope.go:122] "RemoveContainer" containerID="fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"
Nov 01 22:52:20.744935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.744908    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token" pod "6dcabe3e-1f3b-4471-904d-fd934a18a547" (UID: "6dcabe3e-1f3b-4471-904d-fd934a18a547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.763887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.763861    5031 scope.go:122] "RemoveContainer" containerID="fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"
Nov 01 22:52:20.764291 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.764265    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9\": container with ID starting with fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9 not found: ID does not exist" containerID="fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"
Nov 01 22:52:20.764408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.764299    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9"} err="failed to get container status \"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9\": rpc error: code = NotFound desc = could not find container \"fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9\": container with ID starting with fdc41b2f827b7eec672a7e6b6fa44b1da2cb21bb4cbd523dd6e6c1787c530ab9 not found: ID does not exist"
Nov 01 22:52:20.764408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.764319    5031 scope.go:122] "RemoveContainer" containerID="d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"
Nov 01 22:52:20.777341 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.777315    5031 scope.go:122] "RemoveContainer" containerID="d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"
Nov 01 22:52:20.777739 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.777710    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454\": container with ID starting with d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454 not found: ID does not exist" containerID="d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"
Nov 01 22:52:20.777855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.777750    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454"} err="failed to get container status \"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454\": rpc error: code = NotFound desc = could not find container \"d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454\": container with ID starting with d750b9411903262ea511311e3bf62d5156979a5a00c068452f8bdad95ca9f454 not found: ID does not exist"
Nov 01 22:52:20.777855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.777776    5031 scope.go:122] "RemoveContainer" containerID="1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"
Nov 01 22:52:20.789698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.789650    5031 scope.go:122] "RemoveContainer" containerID="1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"
Nov 01 22:52:20.790084 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.790054    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72\": container with ID starting with 1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72 not found: ID does not exist" containerID="1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"
Nov 01 22:52:20.790184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.790096    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72"} err="failed to get container status \"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72\": rpc error: code = NotFound desc = could not find container \"1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72\": container with ID starting with 1289cfde1359d3da2776c366678f181fd823f0b310cb49e847f517ea3bbc3d72 not found: ID does not exist"
Nov 01 22:52:20.790184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.790122    5031 scope.go:122] "RemoveContainer" containerID="68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"
Nov 01 22:52:20.807195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.807157    5031 scope.go:122] "RemoveContainer" containerID="68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"
Nov 01 22:52:20.808786 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.808733    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3\": container with ID starting with 68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3 not found: ID does not exist" containerID="68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"
Nov 01 22:52:20.808953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.808796    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3"} err="failed to get container status \"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3\": rpc error: code = NotFound desc = could not find container \"68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3\": container with ID starting with 68a3c3b7b5daf9065663af7b0e33a6772e06376753724b968e505c165ea7afc3 not found: ID does not exist"
Nov 01 22:52:20.808953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.808821    5031 scope.go:122] "RemoveContainer" containerID="5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"
Nov 01 22:52:20.823189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.823170    5031 scope.go:122] "RemoveContainer" containerID="813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"
Nov 01 22:52:20.833990 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.833970    5031 scope.go:122] "RemoveContainer" containerID="5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"
Nov 01 22:52:20.834309 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.834288    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5\": container with ID starting with 5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5 not found: ID does not exist" containerID="5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"
Nov 01 22:52:20.834389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.834317    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5"} err="failed to get container status \"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5\": rpc error: code = NotFound desc = could not find container \"5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5\": container with ID starting with 5390024022acc664794dfe29988317b3c80ddf4acd7c6623b151e951c426c5d5 not found: ID does not exist"
Nov 01 22:52:20.834389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.834335    5031 scope.go:122] "RemoveContainer" containerID="813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"
Nov 01 22:52:20.834656 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:20.834627    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d\": container with ID starting with 813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d not found: ID does not exist" containerID="813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"
Nov 01 22:52:20.834768 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.834665    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d"} err="failed to get container status \"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d\": rpc error: code = NotFound desc = could not find container \"813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d\": container with ID starting with 813e58e97e8591cca4231cab6d1f60be7fc51211d35716371aa3fd47f6ba158d not found: ID does not exist"
Nov 01 22:52:20.837171 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837145    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.837298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837196    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.837298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837226    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.837298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837272    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.837466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837319    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.837466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837346    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.837466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837380    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.837466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837411    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.837466 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837463    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837496    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837523    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837589    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837626    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837641    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837669    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837698    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.837729 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837725    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837742    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837763    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837770    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837783    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837828    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837881    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837917    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837943    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837973    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.837972    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838004    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838060    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838067    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838106    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838142    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838200    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.838228 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838241    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838284    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838308    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838330    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838388    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838516    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838560    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838634    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838677    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838717    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838751    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838833    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838853    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838877    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838931    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.838967    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\") pod \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\" (UID: \"14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839015    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\") pod \"ae3a252d-49cf-415c-8a86-80e048220cd4\" (UID: \"ae3a252d-49cf-415c-8a86-80e048220cd4\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839046    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839102    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\") pod \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\" (UID: \"d9ddc310-ff4a-407e-b55b-304ffb3fd629\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839144    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\") pod \"7d484274-310f-4149-be9d-6a62c1f8ae93\" (UID: \"7d484274-310f-4149-be9d-6a62c1f8ae93\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839214    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\") pod \"6db83d80-352b-4a54-93e1-427d5aadff08\" (UID: \"6db83d80-352b-4a54-93e1-427d5aadff08\") "
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839237    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839701    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.840036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.839939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840136    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840154    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840286    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840305    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840322    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840337    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3a252d-49cf-415c-8a86-80e048220cd4-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840338    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "kube-api-access-rmnsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840396    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840411    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840424    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840438    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db83d80-352b-4a54-93e1-427d5aadff08-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840453    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f42w7\" (UniqueName: \"kubernetes.io/projected/6dcabe3e-1f3b-4471-904d-fd934a18a547-kube-api-access-f42w7\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840467    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840472    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840480    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840511    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840526    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840557    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcabe3e-1f3b-4471-904d-fd934a18a547-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840571    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840584    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840597    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840727    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "kube-api-access-djt79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.841366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.840780    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.842832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.842807    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.843117 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843018    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.843204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843126    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.843342 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843321    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "kube-api-access-kh6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.843412 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843350    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.843412 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843382    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.843524 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843420    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.843646 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843607    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "kube-api-access-275c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.843744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.843725    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.844758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.844731    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.846290 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846018    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.846290 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846058    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.846437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846350    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.846437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846405    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.846612 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846579    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.846758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846736    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.846900 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846879    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.846959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.846904    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "kube-api-access-sldnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.847107 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.847083    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data" pod "ae3a252d-49cf-415c-8a86-80e048220cd4" (UID: "ae3a252d-49cf-415c-8a86-80e048220cd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.847212 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.847175    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:20.847439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.847417    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.847510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.847423    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:20.847510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.847478    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.848524 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.848501    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy" pod "7d484274-310f-4149-be9d-6a62c1f8ae93" (UID: "7d484274-310f-4149-be9d-6a62c1f8ae93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.848797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.848723    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo" pod "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" (UID: "14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.848797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.848755    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data" pod "d9ddc310-ff4a-407e-b55b-304ffb3fd629" (UID: "d9ddc310-ff4a-407e-b55b-304ffb3fd629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:52:20.849081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.849066    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo" pod "6db83d80-352b-4a54-93e1-427d5aadff08" (UID: "6db83d80-352b-4a54-93e1-427d5aadff08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:52:20.942275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942232    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942275    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942290    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942303    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942316    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6jv\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-kube-api-access-kh6jv\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942329    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942342    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942355    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942368    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275c4\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-kube-api-access-275c4\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942383    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db83d80-352b-4a54-93e1-427d5aadff08-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942396    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942410    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942423    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldnx\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-kube-api-access-sldnx\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942436    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942450    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942463    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942479    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942491    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnsh\" (UniqueName: \"kubernetes.io/projected/6db83d80-352b-4a54-93e1-427d5aadff08-kube-api-access-rmnsh\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.942505 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942505    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djt79\" (UniqueName: \"kubernetes.io/projected/7d484274-310f-4149-be9d-6a62c1f8ae93-kube-api-access-djt79\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942518    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942558    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d484274-310f-4149-be9d-6a62c1f8ae93-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942579    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942595    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942608    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942620    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942632    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3a252d-49cf-415c-8a86-80e048220cd4-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942645    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942657    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3a252d-49cf-415c-8a86-80e048220cd4-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942670    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942684    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d484274-310f-4149-be9d-6a62c1f8ae93-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942696    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db83d80-352b-4a54-93e1-427d5aadff08-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:20.943012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:20.942710    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ddc310-ff4a-407e-b55b-304ffb3fd629-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:21.224039 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.223983    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:52:21.228329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.228293    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-krctp"]
Nov 01 22:52:21.316494 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.316388    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:52:21.319850 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.319820    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-clqr8"]
Nov 01 22:52:21.801889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.801850    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:52:21.808532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.808498    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-xq74c"]
Nov 01 22:52:21.909522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.909465    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:52:21.915842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:21.915805    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-4dnjs"]
Nov 01 22:52:22.019251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.019212    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:52:22.022166 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.022135    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-lrgrf"]
Nov 01 22:52:22.128053 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.127952    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:52:22.132761 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.132723    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-k6slf"]
Nov 01 22:52:22.172969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.172932    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" path="/var/lib/kubelet/pods/185b08a2-1ec2-418f-90b1-43b584d46163/volumes"
Nov 01 22:52:22.173444 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.173430    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" path="/var/lib/kubelet/pods/6db83d80-352b-4a54-93e1-427d5aadff08/volumes"
Nov 01 22:52:22.173886 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.173872    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" path="/var/lib/kubelet/pods/6dcabe3e-1f3b-4471-904d-fd934a18a547/volumes"
Nov 01 22:52:22.174617 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.174603    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" path="/var/lib/kubelet/pods/ae3a252d-49cf-415c-8a86-80e048220cd4/volumes"
Nov 01 22:52:22.175032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.175019    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" path="/var/lib/kubelet/pods/d9ddc310-ff4a-407e-b55b-304ffb3fd629/volumes"
Nov 01 22:52:22.175417 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.175405    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" path="/var/lib/kubelet/pods/fb4b1daa-bf3e-403d-845b-7558ca6165be/volumes"
Nov 01 22:52:22.234829 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.234796    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:52:22.242282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.242249    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-p7sqf"]
Nov 01 22:52:22.345300 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.345260    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:52:22.349616 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:22.349586    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-kzb7h"]
Nov 01 22:52:23.550907 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.550569    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 22:52:23.554961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.554926    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy"
Nov 01 22:52:23.555188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555170    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy"
Nov 01 22:52:23.555324 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555312    5031 cpu_manager.go:414] "containerMap: removing container" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy"
Nov 01 22:52:23.555415 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555405    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy"
Nov 01 22:52:23.555528 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555517    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy"
Nov 01 22:52:23.555648 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555636    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy"
Nov 01 22:52:23.555767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555756    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy"
Nov 01 22:52:23.555895 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.555881    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy"
Nov 01 22:52:23.556029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556018    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy"
Nov 01 22:52:23.556120 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556109    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy"
Nov 01 22:52:23.556221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556211    5031 cpu_manager.go:414] "containerMap: removing container" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy"
Nov 01 22:52:23.556305 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556295    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy"
Nov 01 22:52:23.556396 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556388    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy"
Nov 01 22:52:23.556478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556468    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy"
Nov 01 22:52:23.556599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556588    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.556694 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556683    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.556819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556809    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.556906 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.556896    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.557187 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557173    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.557446 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557430    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db83d80-352b-4a54-93e1-427d5aadff08" containerName="istio-proxy"
Nov 01 22:52:23.557591 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557580    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb4b1daa-bf3e-403d-845b-7558ca6165be" containerName="istio-proxy"
Nov 01 22:52:23.557704 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557695    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcabe3e-1f3b-4471-904d-fd934a18a547" containerName="istio-proxy"
Nov 01 22:52:23.557808 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557798    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" containerName="istio-proxy"
Nov 01 22:52:23.557910 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.557900    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" containerName="istio-proxy"
Nov 01 22:52:23.558036 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.558024    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="185b08a2-1ec2-418f-90b1-43b584d46163" containerName="istio-proxy"
Nov 01 22:52:23.558142 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.558131    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ddc310-ff4a-407e-b55b-304ffb3fd629" containerName="istio-proxy"
Nov 01 22:52:23.569261 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.568999    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.624969 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.624901    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bd7ff8-openshift-default-dockercfg-4pjz7\""
Nov 01 22:52:23.683265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.683238    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 22:52:23.696833 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.696803    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.696912    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.696948    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697176 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697052    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lp5l\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697176 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697157    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697210    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697352 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697313    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697377    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.697465 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.697406    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.762781 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.762747    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 22:52:23.763436 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.763421    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3a252d-49cf-415c-8a86-80e048220cd4" containerName="istio-proxy"
Nov 01 22:52:23.767333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.767312    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.798826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.798788    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.798883    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.798932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.798961    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799073    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmqg\" (UniqueName: \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.799222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799147    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lp5l\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799223    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.799336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799235    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799276    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799485 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799339    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799485 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799421    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.799485 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799451    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.799654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799563    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799578    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799614    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799790 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799679    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799790 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799685    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.799889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.799870    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.801530 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.801465    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.802073 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.802051    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.825204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.825166    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 22:52:23.854254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.854223    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-646881-serving-cert\""
Nov 01 22:52:23.854634 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.854604    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lp5l\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.864000 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.863968    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\") pod \"e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") " pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:23.900913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.900870    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.901154 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.900971    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.901154 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.900998    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.901338 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.901214    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmqg\" (UniqueName: \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.901338 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:23.901228    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-serving-cert: secret "maas-api-e2e-shared-a-w2-646881-serving-cert" not found
Nov 01 22:52:23.901338 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:23.901327    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls podName:f0e6e810-2bcf-4269-9106-7cf61f233eb7 nodeName:}" failed. No retries permitted until 2026-11-01 22:52:24.40130272 +0000 UTC m=+1698.357342562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls") pod "maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" (UID: "f0e6e810-2bcf-4269-9106-7cf61f233eb7") : secret "maas-api-e2e-shared-a-w2-646881-serving-cert" not found
Nov 01 22:52:23.902206 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.902185    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.905004 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.904978    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.958481 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.958438    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmqg\" (UniqueName: \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:23.960928 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:23.960902    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:24.173059 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.172975    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12" path="/var/lib/kubelet/pods/14f52a4a-c87c-4ef0-8a4a-7b728c9e3c12/volumes"
Nov 01 22:52:24.173495 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.173479    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d484274-310f-4149-be9d-6a62c1f8ae93" path="/var/lib/kubelet/pods/7d484274-310f-4149-be9d-6a62c1f8ae93/volumes"
Nov 01 22:52:24.390942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.390907    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 22:52:24.396018 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:24.395987    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d84ff32_5ab1_4588_b7de_a97ae8f70385.slice/crio-1e222b54192b4bc10ac146fe189bb8454c89c3880e71db9b6c92ee97994fcfb4 WatchSource:0}: Error finding container 1e222b54192b4bc10ac146fe189bb8454c89c3880e71db9b6c92ee97994fcfb4: Status 404 returned error can't find the container with id 1e222b54192b4bc10ac146fe189bb8454c89c3880e71db9b6c92ee97994fcfb4
Nov 01 22:52:24.399184 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.399144    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:52:24.399274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.399219    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:52:24.399274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.399249    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:52:24.409265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.409228    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:24.411960 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.411933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:24.681221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.681179    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:24.773759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.773695    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerStarted","Data":"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"}
Nov 01 22:52:24.773974 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.773769    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerStarted","Data":"1e222b54192b4bc10ac146fe189bb8454c89c3880e71db9b6c92ee97994fcfb4"}
Nov 01 22:52:24.887038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.886975    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podStartSLOduration=1.886959182 podStartE2EDuration="1.886959182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:24.884066201 +0000 UTC m=+1698.840106046" watchObservedRunningTime="2026-11-01 22:52:24.886959182 +0000 UTC m=+1698.842999026"
Nov 01 22:52:24.961358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:24.961264    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:25.062028 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:25.061702    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 22:52:25.782894 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:25.782858    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" event={"ID":"f0e6e810-2bcf-4269-9106-7cf61f233eb7","Type":"ContainerStarted","Data":"67f9c2566d373e2565ac0cc1d835d47492699d966bda326fd9ce80a0c187502d"}
Nov 01 22:52:25.971842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:25.971801    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:26.799355 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:26.799310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" event={"ID":"f0e6e810-2bcf-4269-9106-7cf61f233eb7","Type":"ContainerStarted","Data":"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"}
Nov 01 22:52:26.799878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:26.799804    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:26.799878 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:26.799848    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:26.801631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:26.801600    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:52:26.909470 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:26.909416    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" podStartSLOduration=3.021909644 podStartE2EDuration="3.909401377s" totalImagesPullingTime="887.491733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:26.907634347 +0000 UTC m=+1700.863674181" watchObservedRunningTime="2026-11-01 22:52:26.909401377 +0000 UTC m=+1700.865441221"
Nov 01 22:52:32.809954 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.809916    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:52:32.814693 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.814667    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 22:52:32.814875 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.814758    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:32.916217 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.916175    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:52:32.919897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.916571    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bd7ff8-dockercfg-ftgzb\""
Nov 01 22:52:32.920128 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.916620    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bd7ff8\""
Nov 01 22:52:32.963563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.939428    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:32.963563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.959819    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprf8\" (UniqueName: \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:32.970599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.965956    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:52:32.972087 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:32.972058    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.023562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.022907    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:52:33.073286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.073057    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:33.088600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.084438    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vts5\" (UniqueName: \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.088600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.084633    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprf8\" (UniqueName: \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:33.088600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.084984    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.088600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.087027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:33.139444 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.139414    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprf8\" (UniqueName: \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\") pod \"payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:33.186599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.186513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vts5\" (UniqueName: \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.186780 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.186653    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.187435 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.187413    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.241822 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.241794    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vts5\" (UniqueName: \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\") pod \"payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.307600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.307560    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.429670 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.429634    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:33.695147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.695066    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:52:33.700858 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:33.700827    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8ce478_dcb8_4fec_9db3_318ca987fc2a.slice/crio-38e3ff294748f6d05cf351c6228fc05b2ea591fff5f17a449156f89511a965c1 WatchSource:0}: Error finding container 38e3ff294748f6d05cf351c6228fc05b2ea591fff5f17a449156f89511a965c1: Status 404 returned error can't find the container with id 38e3ff294748f6d05cf351c6228fc05b2ea591fff5f17a449156f89511a965c1
Nov 01 22:52:33.823212 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.823174    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:52:33.831863 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:33.831827    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8de0329_3d28_42f1_97c4_e88b85452011.slice/crio-8d465fd07491670d006f8b4bc53853ec25ec5d2110d51e2c72afa52fc3669d23 WatchSource:0}: Error finding container 8d465fd07491670d006f8b4bc53853ec25ec5d2110d51e2c72afa52fc3669d23: Status 404 returned error can't find the container with id 8d465fd07491670d006f8b4bc53853ec25ec5d2110d51e2c72afa52fc3669d23
Nov 01 22:52:33.845556 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.845487    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" event={"ID":"f8de0329-3d28-42f1-97c4-e88b85452011","Type":"ContainerStarted","Data":"8d465fd07491670d006f8b4bc53853ec25ec5d2110d51e2c72afa52fc3669d23"}
Nov 01 22:52:33.847467 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.847433    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" event={"ID":"0b8ce478-dcb8-4fec-9db3-318ca987fc2a","Type":"ContainerStarted","Data":"ae6ce2580a131e66a4ceda691343907c6599c72f24d8a06d082407bb27617645"}
Nov 01 22:52:33.847636 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.847492    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" event={"ID":"0b8ce478-dcb8-4fec-9db3-318ca987fc2a","Type":"ContainerStarted","Data":"38e3ff294748f6d05cf351c6228fc05b2ea591fff5f17a449156f89511a965c1"}
Nov 01 22:52:33.847696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.847657    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:33.962014 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:33.961888    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" podStartSLOduration=1.96183188 podStartE2EDuration="1.96183188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:33.958443542 +0000 UTC m=+1707.914483386" watchObservedRunningTime="2026-11-01 22:52:33.96183188 +0000 UTC m=+1707.917871723"
Nov 01 22:52:34.317782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.317747    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 22:52:34.322272 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.322246    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.370975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.370946    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-8bac2f-serving-cert\""
Nov 01 22:52:34.382024 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.381989    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 22:52:34.407150 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.407105    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.407331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.407189    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.407424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.407400    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfjl\" (UniqueName: \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.407520 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.407460    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.508449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.508412    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.508621 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.508486    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.508621 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.508614    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfjl\" (UniqueName: \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.508716 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:34.508636    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-serving-cert: secret "maas-api-e2e-shared-a-w4-8bac2f-serving-cert" not found
Nov 01 22:52:34.508716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.508650    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.508788 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:34.508737    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls podName:d4bcb313-b6e0-4fe2-a82e-900e3f715dc6 nodeName:}" failed. No retries permitted until 2026-11-01 22:52:35.008700417 +0000 UTC m=+1708.964740250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls") pod "maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" (UID: "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6") : secret "maas-api-e2e-shared-a-w4-8bac2f-serving-cert" not found
Nov 01 22:52:34.509426 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.509402    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.511150 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.511127    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.567848 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.566186    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfjl\" (UniqueName: \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:34.855579 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.855435    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" event={"ID":"f8de0329-3d28-42f1-97c4-e88b85452011","Type":"ContainerStarted","Data":"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"}
Nov 01 22:52:34.856049 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.855716    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:34.964249 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:34.964190    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" podStartSLOduration=2.96417383 podStartE2EDuration="2.96417383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:34.961106602 +0000 UTC m=+1708.917146447" watchObservedRunningTime="2026-11-01 22:52:34.96417383 +0000 UTC m=+1708.920213675"
Nov 01 22:52:35.016308 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:35.016273    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:35.018975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:35.018945    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:35.237005 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:35.236893    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:35.629403 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:35.629364    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 22:52:35.633961 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:35.633931    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bcb313_b6e0_4fe2_a82e_900e3f715dc6.slice/crio-d619977251bd6362b044696c1a953209fc8f472248ee6da589dd7a334d91a054 WatchSource:0}: Error finding container d619977251bd6362b044696c1a953209fc8f472248ee6da589dd7a334d91a054: Status 404 returned error can't find the container with id d619977251bd6362b044696c1a953209fc8f472248ee6da589dd7a334d91a054
Nov 01 22:52:35.863884 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:35.863840    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" event={"ID":"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6","Type":"ContainerStarted","Data":"d619977251bd6362b044696c1a953209fc8f472248ee6da589dd7a334d91a054"}
Nov 01 22:52:36.874183 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:36.874078    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" event={"ID":"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6","Type":"ContainerStarted","Data":"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"}
Nov 01 22:52:36.874696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:36.874180    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:36.984684 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:36.984624    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" podStartSLOduration=2.058222474 podStartE2EDuration="2.984605981s" totalImagesPullingTime="926.383507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:36.980895201 +0000 UTC m=+1710.936935046" watchObservedRunningTime="2026-11-01 22:52:36.984605981 +0000 UTC m=+1710.940645825"
Nov 01 22:52:40.651564 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:40.645586    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:52:42.888124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:42.888096    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 22:52:44.858925 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:44.858898    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:52:44.894633 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:44.894596    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 22:52:44.899327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:44.899294    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:44.948241 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:44.948213    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-2b0866-serving-cert\""
Nov 01 22:52:44.957787 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:44.957758    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 22:52:45.096620 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.096523    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.096816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.096688    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.096881 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.096863    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.097029 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.097004    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj42p\" (UniqueName: \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.198410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.198309    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.198627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.198421    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.198627 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:45.198481    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-serving-cert: secret "maas-api-e2e-shared-b-w0-2b0866-serving-cert" not found
Nov 01 22:52:45.198627 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.198508    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj42p\" (UniqueName: \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.198627 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:52:45.198574    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls podName:54bef2ed-5d69-413b-8ab2-c6d127797e70 nodeName:}" failed. No retries permitted until 2026-11-01 22:52:45.69855119 +0000 UTC m=+1719.654591025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls") pod "maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" (UID: "54bef2ed-5d69-413b-8ab2-c6d127797e70") : secret "maas-api-e2e-shared-b-w0-2b0866-serving-cert" not found
Nov 01 22:52:45.198854 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.198671    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.199477 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.199456    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.201360 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.201326    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.253251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.253222    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj42p\" (UniqueName: \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.708381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.708340    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.711031 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.711003    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.814786 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.814751    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:45.867957 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:45.867931    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:52:46.194260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:46.194216    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 22:52:46.199395 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:46.199364    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bef2ed_5d69_413b_8ab2_c6d127797e70.slice/crio-8f6959e3290b484c2c33c396fd1f78b6a55b45a3dca388cbcde8d8e1e8b29e8a WatchSource:0}: Error finding container 8f6959e3290b484c2c33c396fd1f78b6a55b45a3dca388cbcde8d8e1e8b29e8a: Status 404 returned error can't find the container with id 8f6959e3290b484c2c33c396fd1f78b6a55b45a3dca388cbcde8d8e1e8b29e8a
Nov 01 22:52:46.969383 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:46.969345    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" event={"ID":"54bef2ed-5d69-413b-8ab2-c6d127797e70","Type":"ContainerStarted","Data":"8f6959e3290b484c2c33c396fd1f78b6a55b45a3dca388cbcde8d8e1e8b29e8a"}
Nov 01 22:52:47.977299 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:47.977262    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" event={"ID":"54bef2ed-5d69-413b-8ab2-c6d127797e70","Type":"ContainerStarted","Data":"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"}
Nov 01 22:52:47.977771 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:47.977327    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:52:48.086510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.086444    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" podStartSLOduration=3.12750595 podStartE2EDuration="4.086423799s" totalImagesPullingTime="958.917849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:48.085755062 +0000 UTC m=+1722.041794906" watchObservedRunningTime="2026-11-01 22:52:48.086423799 +0000 UTC m=+1722.042463646"
Nov 01 22:52:48.744350 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.744319    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:52:48.748752 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.748727    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:48.797557 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.797508    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-f5b935-dockercfg-nm9wm\""
Nov 01 22:52:48.797557 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.797556    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-f5b935\""
Nov 01 22:52:48.809612 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.809571    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:52:48.877020 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.876973    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:48.877314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.877288    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvkf5\" (UniqueName: \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:48.894886 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.894846    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:52:48.901907 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.901853    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:48.948032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.947995    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:52:48.992776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.992738    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:48.993275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.992804    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:48.993275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.992849    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvkf5\" (UniqueName: \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:48.993275 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.992961    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcl2\" (UniqueName: \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:48.994130 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:48.994092    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:49.048936 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.048895    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvkf5\" (UniqueName: \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\") pod \"payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:49.068942 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.068882    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:49.096362 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.094970    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcl2\" (UniqueName: \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:49.096362 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.095294    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:49.096773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.096684    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:49.148984 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.148929    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcl2\" (UniqueName: \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\") pod \"payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:49.220618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.220576    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:49.476295 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.476213    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:52:49.480282 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:49.480252    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e96df97_353f_4988_aba1_d95289917201.slice/crio-ebd335d047d0090ef74f66f992b37be6eb042bc5561e3f68a7160f56905bd591 WatchSource:0}: Error finding container ebd335d047d0090ef74f66f992b37be6eb042bc5561e3f68a7160f56905bd591: Status 404 returned error can't find the container with id ebd335d047d0090ef74f66f992b37be6eb042bc5561e3f68a7160f56905bd591
Nov 01 22:52:49.628654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:49.628621    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:52:49.637198 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:52:49.637171    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1249469_a0c5_42e6_88a1_d88edec812e2.slice/crio-4fa6671c4c1ef2359e8b8c254598a9a685f7d75c4c1a20e601c0e3c4ed11da41 WatchSource:0}: Error finding container 4fa6671c4c1ef2359e8b8c254598a9a685f7d75c4c1a20e601c0e3c4ed11da41: Status 404 returned error can't find the container with id 4fa6671c4c1ef2359e8b8c254598a9a685f7d75c4c1a20e601c0e3c4ed11da41
Nov 01 22:52:50.011888 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.011849    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" event={"ID":"d1249469-a0c5-42e6-88a1-d88edec812e2","Type":"ContainerStarted","Data":"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"}
Nov 01 22:52:50.012314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.011906    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" event={"ID":"d1249469-a0c5-42e6-88a1-d88edec812e2","Type":"ContainerStarted","Data":"4fa6671c4c1ef2359e8b8c254598a9a685f7d75c4c1a20e601c0e3c4ed11da41"}
Nov 01 22:52:50.012314 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.011936    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:52:50.013722 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.013691    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" event={"ID":"8e96df97-353f-4988-aba1-d95289917201","Type":"ContainerStarted","Data":"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"}
Nov 01 22:52:50.013849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.013745    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" event={"ID":"8e96df97-353f-4988-aba1-d95289917201","Type":"ContainerStarted","Data":"ebd335d047d0090ef74f66f992b37be6eb042bc5561e3f68a7160f56905bd591"}
Nov 01 22:52:50.013849 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.013766    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:52:50.122167 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.122113    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" podStartSLOduration=2.122034998 podStartE2EDuration="2.122034998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:50.121298115 +0000 UTC m=+1724.077337960" watchObservedRunningTime="2026-11-01 22:52:50.122034998 +0000 UTC m=+1724.078074843"
Nov 01 22:52:50.233346 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:50.233281    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" podStartSLOduration=2.233263611 podStartE2EDuration="2.233263611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:50.229845434 +0000 UTC m=+1724.185885278" watchObservedRunningTime="2026-11-01 22:52:50.233263611 +0000 UTC m=+1724.189303464"
Nov 01 22:52:54.009254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:52:54.009225    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 22:53:01.024487 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:01.024452    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:53:01.024972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:01.024720    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:53:10.627307 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:53:10.627267    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:53:40.632094 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:53:40.632053    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:53:47.901682 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:47.901647    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 22:53:47.906361 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:47.906337    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:47.937532 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:47.937482    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:47.937795 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:47.937636    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5dp\" (UniqueName: \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.001993 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.001955    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8cc47a-dockercfg-8gq6d\""
Nov 01 22:53:48.007931 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.007890    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8cc47a\""
Nov 01 22:53:48.010551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.010496    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 22:53:48.042160 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.041620    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.042160 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.041807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5dp\" (UniqueName: \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.043251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.043214    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.097282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.097246    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5dp\" (UniqueName: \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\") pod \"payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.226756 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.226665    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:48.606143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:48.606107    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 22:53:48.611131 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:53:48.611102    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c8b445_4220_4796_8df4_bfc9b30bd951.slice/crio-638a1614b738a2950b5eec1a7c597a68e5c5cac4f8f1b53d4e7341246ed2feeb WatchSource:0}: Error finding container 638a1614b738a2950b5eec1a7c597a68e5c5cac4f8f1b53d4e7341246ed2feeb: Status 404 returned error can't find the container with id 638a1614b738a2950b5eec1a7c597a68e5c5cac4f8f1b53d4e7341246ed2feeb
Nov 01 22:53:49.506180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:49.506130    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" event={"ID":"52c8b445-4220-4796-8df4-bfc9b30bd951","Type":"ContainerStarted","Data":"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"}
Nov 01 22:53:49.506180 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:49.506188    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" event={"ID":"52c8b445-4220-4796-8df4-bfc9b30bd951","Type":"ContainerStarted","Data":"638a1614b738a2950b5eec1a7c597a68e5c5cac4f8f1b53d4e7341246ed2feeb"}
Nov 01 22:53:49.506683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:49.506291    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:53:49.615302 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:49.615247    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" podStartSLOduration=2.615230508 podStartE2EDuration="2.615230508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:49.612978542 +0000 UTC m=+1783.569018387" watchObservedRunningTime="2026-11-01 22:53:49.615230508 +0000 UTC m=+1783.571270352"
Nov 01 22:53:56.762600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.762174    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 22:53:56.770564 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.767438    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.806665 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.806632    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 22:53:56.817604 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.817065    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-f5b935-serving-cert\""
Nov 01 22:53:56.887863 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.887823    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.888081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.887901    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.888081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.888026    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6466\" (UniqueName: \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.888205 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.888148    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.990186 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.990154    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6466\" (UniqueName: \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.990410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.990297    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.990472 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:53:56.990451    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-serving-cert: secret "maas-api-e2e-shared-a-w6-f5b935-serving-cert" not found
Nov 01 22:53:56.990472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.990458    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.990584 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:53:56.990532    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls podName:c094b927-ca3e-4e73-bd42-2b38968d7ce4 nodeName:}" failed. No retries permitted until 2026-11-01 22:53:57.490510405 +0000 UTC m=+1791.446550230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls") pod "maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" (UID: "c094b927-ca3e-4e73-bd42-2b38968d7ce4") : secret "maas-api-e2e-shared-a-w6-f5b935-serving-cert" not found
Nov 01 22:53:56.990645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.990599    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.991143 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.991118    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:56.993630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:56.993601    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:57.046641 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:57.046506    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6466\" (UniqueName: \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:57.498369 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:57.498271    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:57.501012 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:57.500986    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:57.701734 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:57.701700    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:58.076599 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:58.076562    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 22:53:58.083888 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:53:58.083856    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc094b927_ca3e_4e73_bd42_2b38968d7ce4.slice/crio-259fa04326a63425fc4cf4cb767b270795772035aa4bf4cc4cc7665d576312e1 WatchSource:0}: Error finding container 259fa04326a63425fc4cf4cb767b270795772035aa4bf4cc4cc7665d576312e1: Status 404 returned error can't find the container with id 259fa04326a63425fc4cf4cb767b270795772035aa4bf4cc4cc7665d576312e1
Nov 01 22:53:58.572708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:58.572676    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" event={"ID":"c094b927-ca3e-4e73-bd42-2b38968d7ce4","Type":"ContainerStarted","Data":"259fa04326a63425fc4cf4cb767b270795772035aa4bf4cc4cc7665d576312e1"}
Nov 01 22:53:59.581280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.581238    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" event={"ID":"c094b927-ca3e-4e73-bd42-2b38968d7ce4","Type":"ContainerStarted","Data":"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"}
Nov 01 22:53:59.581821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.581304    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:53:59.681284 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.681240    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 22:53:59.686524 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.686495    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.727469 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.727395    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" podStartSLOduration=2.799927774 podStartE2EDuration="3.727367202s" totalImagesPullingTime="927.439428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:59.690384384 +0000 UTC m=+1793.646424245" watchObservedRunningTime="2026-11-01 22:53:59.727367202 +0000 UTC m=+1793.683407046"
Nov 01 22:53:59.735449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.735412    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8bac2f-openshift-default-dockercfg-dvrdf\""
Nov 01 22:53:59.771064 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.771018    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 22:53:59.832203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832102    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832155    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832294    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832584 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832520    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832732 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832653    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832732 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832693    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832839    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832915 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832884    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pbn\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.832991 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.832971    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934355    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934391 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934397    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pbn\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934424    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934471    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934504    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.934765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934577    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.935047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934772    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.935047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934834    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.935047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934877    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.935047 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.934878    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.935367 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.935225    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.936419 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.936391    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.937101 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.937077    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.937421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.937397    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.938871 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.938848    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.939022 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.938950    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.989204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.989174    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:53:59.989424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:53:59.989399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pbn\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:00.004623 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.004592    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:00.432278 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.432236    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 22:54:00.473838 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:54:00.473803    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755e265e_0fc2_40d5_a91a_62c293471873.slice/crio-283455c983352435be50f87e4cd1a6d57da863db6ca055b2ac9e05252c4e515c WatchSource:0}: Error finding container 283455c983352435be50f87e4cd1a6d57da863db6ca055b2ac9e05252c4e515c: Status 404 returned error can't find the container with id 283455c983352435be50f87e4cd1a6d57da863db6ca055b2ac9e05252c4e515c
Nov 01 22:54:00.476680 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.476638    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:00.476785 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.476716    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:00.476785 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.476747    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:00.516026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.515991    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 22:54:00.590265 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.590229    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" event={"ID":"755e265e-0fc2-40d5-a91a-62c293471873","Type":"ContainerStarted","Data":"068d9ad0a289cfcf8aa9edd09ef9e9b70da31e8a5b3edb7a3f5718612075882a"}
Nov 01 22:54:00.590718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.590292    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" event={"ID":"755e265e-0fc2-40d5-a91a-62c293471873","Type":"ContainerStarted","Data":"283455c983352435be50f87e4cd1a6d57da863db6ca055b2ac9e05252c4e515c"}
Nov 01 22:54:00.742793 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:00.742654    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podStartSLOduration=1.7426292399999999 podStartE2EDuration="1.74262924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:00.732506984 +0000 UTC m=+1794.688546828" watchObservedRunningTime="2026-11-01 22:54:00.74262924 +0000 UTC m=+1794.698669083"
Nov 01 22:54:01.005483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:01.005390    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:01.007190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:01.007164    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:01.007309 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:01.007214    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:02.005312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:02.005274    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:02.005774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:02.005334    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:03.005316 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:03.005281    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:03.005749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:03.005335    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:04.005041 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:04.005004    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:04.005225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:04.005070    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:04.021189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:04.021148    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:54:04.021653 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:04.021381    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" containerID="cri-o://8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6" gracePeriod=30
Nov 01 22:54:05.005151 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:05.005114    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:05.005318 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:05.005182    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:05.596642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:05.596612    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 22:54:06.016190 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.016100    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:06.016430 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.016396    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:06.258561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.258506    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 22:54:06.271933 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.271847    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.319109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.319072    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 22:54:06.321197 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.321168    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-661e9f\""
Nov 01 22:54:06.321399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.321174    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-661e9f-dockercfg-msvbs\""
Nov 01 22:54:06.394566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.371404    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjj7\" (UniqueName: \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.505418 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.504977    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.611959 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.611922    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.612961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.612932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjj7\" (UniqueName: \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.613276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.613246    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.683972 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.683830    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjj7\" (UniqueName: \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\") pod \"payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:06.894650 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:06.894529    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:07.005484 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.005447    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 22:54:07.005786 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.005523    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 22:54:07.307915 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.307876    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 22:54:07.650001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.649914    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" event={"ID":"95c5c865-ba7b-484d-9550-d1d181ff6ef4","Type":"ContainerStarted","Data":"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"}
Nov 01 22:54:07.650001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.649968    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" event={"ID":"95c5c865-ba7b-484d-9550-d1d181ff6ef4","Type":"ContainerStarted","Data":"1b65d64cbbcd336c35aec562d2adb2f39a9bd0c69d6a5417b00a5f6c9d530b1f"}
Nov 01 22:54:07.650439 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.650096    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:07.781600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:07.768606    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" podStartSLOduration=1.7685853219999998 podStartE2EDuration="1.768585322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:07.768097395 +0000 UTC m=+1801.724137240" watchObservedRunningTime="2026-11-01 22:54:07.768585322 +0000 UTC m=+1801.724625166"
Nov 01 22:54:08.010829 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:08.010747    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:08.011247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:08.011228    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:08.012137 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:08.012117    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 22:54:08.740746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:08.740707    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:08.741147 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:08.740786    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:09.401746 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.401720    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:54:09.486062 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486025    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486084    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486161    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486271    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486443 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486372    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.486443 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486394    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486575 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486506    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486480    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.486638 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486605    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486642    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486643    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:09.486742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486681    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.486742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486694    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\") pod \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\" (UID: \"448a2604-d3d3-4ff1-b096-7ee993f98dcb\") "
Nov 01 22:54:09.486977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.486955    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.487612 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.487587    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.487612 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.487615    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.487797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.487628    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.487797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.487658    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.487797 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.487667    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.489051 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.489026    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:09.489241 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.489223    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.489596 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.489577    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "kube-api-access-kxs2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.489659 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.489612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token" pod "448a2604-d3d3-4ff1-b096-7ee993f98dcb" (UID: "448a2604-d3d3-4ff1-b096-7ee993f98dcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.589778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.589294    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxs2c\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-kube-api-access-kxs2c\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.589778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.589323    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.589778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.589332    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.589778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.589341    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/448a2604-d3d3-4ff1-b096-7ee993f98dcb-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.691798 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.691762    5031 generic.go:356] "Generic (PLEG): container finished" podID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerID="8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6" exitCode=0
Nov 01 22:54:09.691977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.691798    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" event={"ID":"448a2604-d3d3-4ff1-b096-7ee993f98dcb","Type":"ContainerDied","Data":"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"}
Nov 01 22:54:09.691977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.691809    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"
Nov 01 22:54:09.691977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.691864    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w" event={"ID":"448a2604-d3d3-4ff1-b096-7ee993f98dcb","Type":"ContainerDied","Data":"15d0e4378c34b1112eb43404b22b9aa62e5276e54128ec86c827625a036cca67"}
Nov 01 22:54:09.691977 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.691892    5031 scope.go:122] "RemoveContainer" containerID="8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"
Nov 01 22:54:09.707917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.707892    5031 scope.go:122] "RemoveContainer" containerID="8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"
Nov 01 22:54:09.708250 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:54:09.708233    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6\": container with ID starting with 8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6 not found: ID does not exist" containerID="8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"
Nov 01 22:54:09.708293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.708261    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6"} err="failed to get container status \"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6\": rpc error: code = NotFound desc = could not find container \"8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6\": container with ID starting with 8d382ac04b49342e61a8cdc85ab8c304b68f88c984a92f13a1863c39da8bf3c6 not found: ID does not exist"
Nov 01 22:54:09.849033 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.848937    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:54:09.869615 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:09.853929    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bgt8w"]
Nov 01 22:54:10.185748 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:10.185661    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" path="/var/lib/kubelet/pods/448a2604-d3d3-4ff1-b096-7ee993f98dcb/volumes"
Nov 01 22:54:10.679185 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:54:10.679148    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:54:16.754866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.754768    5031 generic.go:356] "Generic (PLEG): container finished" podID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerID="794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8" exitCode=0
Nov 01 22:54:16.754866 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.754850    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerDied","Data":"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"}
Nov 01 22:54:16.755445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.755313    5031 scope.go:122] "RemoveContainer" containerID="794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"
Nov 01 22:54:16.757198 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.757161    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:16.757328 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.757256    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:16.757328 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:16.757308    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:17.786889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:17.786855    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerStarted","Data":"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"}
Nov 01 22:54:17.979075 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:17.979029    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:54:17.979282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:17.979172    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:17.979282 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:17.979235    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:18.684490 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:18.684354    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 22:54:18.977628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:18.977000    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:18.977628 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:18.977072    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:19.978815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:19.978766    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:19.979358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:19.978850    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:20.972566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:20.965011    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:20.972566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:20.965076    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:21.979594 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:21.978861    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:21.979594 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:21.978912    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:22.961847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:22.961813    5031 patch_prober.go:29] interesting pod/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 22:54:22.962040 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:22.961871    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 22:54:23.966631 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:23.966600    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:54:23.967056 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:23.967003    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:54:23.967865 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:23.967849    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 22:54:40.627070 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:54:40.627026    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:54:45.079791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.079753    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 22:54:45.080437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.080420    5031 cpu_manager.go:414] "containerMap: removing container" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy"
Nov 01 22:54:45.080437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.080437    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy"
Nov 01 22:54:45.080573 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.080549    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="448a2604-d3d3-4ff1-b096-7ee993f98dcb" containerName="istio-proxy"
Nov 01 22:54:45.085421 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.085399    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.125437 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.125396    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 22:54:45.129899 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.129870    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.188507 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.188469    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 22:54:45.216826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.216783    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.216856    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98tx\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.216905    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.216952    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217032 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.216994    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q42z\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217252 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217052    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217252 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217100    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217252 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217140    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217255    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217310    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217341    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217406 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217382    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217427    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217505    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217555    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.217655 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217593    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217714    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.217847 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.217762    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.246929 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.246893    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 22:54:45.319173 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319120    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.319375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319184    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319210    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319263    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319292    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.319375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319351    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319376    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319404    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319499    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319564    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319615    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319673    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98tx\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.319731 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319723    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.320068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319777    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.320068 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.319816    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q42z\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.320333 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.320302    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.320428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.320367    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.320428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.320401    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321137    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321138    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321688    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321749    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321926    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.322195 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.321972    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.323223 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.323191    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.323488 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.323463    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.324055 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.324027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.324156 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.324033    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.325203 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.325170    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.326132 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.326105    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.326905 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.326882    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.326983 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.326949    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.374262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.374171    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q42z\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.374522 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.374498    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98tx\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.374600 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.374582    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.374726 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.374705    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.401483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.401453    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:45.445109 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.445077    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:45.834736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.834348    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 22:54:45.837869 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:54:45.837840    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8202dac_7445_45b8_91f3_afdac802965d.slice/crio-b7b44644eb4b41c7aac364c5a897ba92d84b2653f97d1765463afc3220224cf7 WatchSource:0}: Error finding container b7b44644eb4b41c7aac364c5a897ba92d84b2653f97d1765463afc3220224cf7: Status 404 returned error can't find the container with id b7b44644eb4b41c7aac364c5a897ba92d84b2653f97d1765463afc3220224cf7
Nov 01 22:54:45.840668 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.840627    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:45.840773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.840703    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:45.840773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.840735    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:45.890834 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.890800    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 22:54:45.894505 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:54:45.894476    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b276892_0640_470f_ba01_70d5f06015ad.slice/crio-0ce108c9ae716713b774dfeb77e094799bebe169c9b7356c4b68051ad8ec6276 WatchSource:0}: Error finding container 0ce108c9ae716713b774dfeb77e094799bebe169c9b7356c4b68051ad8ec6276: Status 404 returned error can't find the container with id 0ce108c9ae716713b774dfeb77e094799bebe169c9b7356c4b68051ad8ec6276
Nov 01 22:54:45.897616 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.897506    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:45.897754 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.897646    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:45.897754 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:45.897700    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:54:46.017964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.017877    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" event={"ID":"c8202dac-7445-45b8-91f3-afdac802965d","Type":"ContainerStarted","Data":"79cf2e6d9085f8ebfb3abcede140ad03e67c87145a9b11f79c23230f64ebd6cd"}
Nov 01 22:54:46.017964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.017937    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" event={"ID":"c8202dac-7445-45b8-91f3-afdac802965d","Type":"ContainerStarted","Data":"b7b44644eb4b41c7aac364c5a897ba92d84b2653f97d1765463afc3220224cf7"}
Nov 01 22:54:46.020812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.020777    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" event={"ID":"9b276892-0640-470f-ba01-70d5f06015ad","Type":"ContainerStarted","Data":"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"}
Nov 01 22:54:46.021022 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.020827    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" event={"ID":"9b276892-0640-470f-ba01-70d5f06015ad","Type":"ContainerStarted","Data":"0ce108c9ae716713b774dfeb77e094799bebe169c9b7356c4b68051ad8ec6276"}
Nov 01 22:54:46.134326 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.134216    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podStartSLOduration=1.134194886 podStartE2EDuration="1.134194886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:46.130508034 +0000 UTC m=+1840.086547894" watchObservedRunningTime="2026-11-01 22:54:46.134194886 +0000 UTC m=+1840.090234732"
Nov 01 22:54:46.244262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.244207    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podStartSLOduration=1.244183646 podStartE2EDuration="1.244183646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:46.239922353 +0000 UTC m=+1840.195962197" watchObservedRunningTime="2026-11-01 22:54:46.244183646 +0000 UTC m=+1840.200223492"
Nov 01 22:54:46.402500 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.402412    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:46.404766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.404725    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:46.405053 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.405019    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:46.445912 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.445870    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:46.447897 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.447869    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:46.448045 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:46.447942    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:47.403019 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:47.402977    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:47.403692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:47.403650    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:47.446451 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:47.446416    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:47.446654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:47.446470    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:48.411855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:48.411209    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:48.411855 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:48.411284    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:48.445700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:48.445665    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:48.445883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:48.445719    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:49.403574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:49.403059    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:49.403574 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:49.403126    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:49.446431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:49.446387    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:49.446851 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:49.446451    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:50.407566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:50.406019    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:50.407566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:50.406082    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:50.445584 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:50.445529    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:50.445767 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:50.445602    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:51.411561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:51.410498    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 22:54:51.411561 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:51.410581    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 22:54:51.446257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:51.446217    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 22:54:51.446453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:51.446275    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 22:54:52.408842 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.408807    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:52.409358 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.409337    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:52.410329 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.410305    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 22:54:52.451393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.451365    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:52.451821 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.451755    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:54:52.452673 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:54:52.452652    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 22:55:10.628115 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:55:10.628074    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:55:20.191690 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.191623    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 22:55:20.201065 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.201039    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.249883 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.249849    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-661e9f-serving-cert\""
Nov 01 22:55:20.300238 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.300202    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 22:55:20.311618 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.311579    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.311810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.311627    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.311810 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.311688    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.311898 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.311826    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xp96\" (UniqueName: \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.413479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.413445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp96\" (UniqueName: \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.413697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.413573    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.413697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.413607    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.413697 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.413656    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.413918 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:55:20.413798    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-serving-cert: secret "maas-api-e2e-shared-b-w4-661e9f-serving-cert" not found
Nov 01 22:55:20.413918 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:55:20.413886    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls podName:cc6c6195-bd41-45b2-8ece-379c55f3f0a7 nodeName:}" failed. No retries permitted until 2026-11-01 22:55:20.91387141 +0000 UTC m=+1874.869911235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls") pod "maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" (UID: "cc6c6195-bd41-45b2-8ece-379c55f3f0a7") : secret "maas-api-e2e-shared-b-w4-661e9f-serving-cert" not found
Nov 01 22:55:20.414383 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.414359    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.416511 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.416484    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.474115 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.474027    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp96\" (UniqueName: \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.924936 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.924892    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:20.927716 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:20.927687    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:21.130774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:21.130726    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:21.512077 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:21.511987    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 22:55:21.517095 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:55:21.517069    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c6195_bd41_45b2_8ece_379c55f3f0a7.slice/crio-abfb355a32d27a9ffa5cd4dd419592864d2f759d5f50c03129c327e440ebd095 WatchSource:0}: Error finding container abfb355a32d27a9ffa5cd4dd419592864d2f759d5f50c03129c327e440ebd095: Status 404 returned error can't find the container with id abfb355a32d27a9ffa5cd4dd419592864d2f759d5f50c03129c327e440ebd095
Nov 01 22:55:22.392492 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:22.392459    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" event={"ID":"cc6c6195-bd41-45b2-8ece-379c55f3f0a7","Type":"ContainerStarted","Data":"abfb355a32d27a9ffa5cd4dd419592864d2f759d5f50c03129c327e440ebd095"}
Nov 01 22:55:23.401389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:23.401354    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" event={"ID":"cc6c6195-bd41-45b2-8ece-379c55f3f0a7","Type":"ContainerStarted","Data":"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"}
Nov 01 22:55:23.401913 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:23.401439    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:23.562889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:23.562824    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" podStartSLOduration=2.744028611 podStartE2EDuration="3.562803277s" totalImagesPullingTime="818.774666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:23.562517907 +0000 UTC m=+1877.518557744" watchObservedRunningTime="2026-11-01 22:55:23.562803277 +0000 UTC m=+1877.518843123"
Nov 01 22:55:29.416502 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:29.416471    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 22:55:35.061737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.061696    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 22:55:35.067757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.067723    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.166765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.165222    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-103369-dockercfg-srw7f\""
Nov 01 22:55:35.166765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.165336    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-103369\""
Nov 01 22:55:35.167982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.167951    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.168425 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.168374    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgk8t\" (UniqueName: \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.171779 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.171751    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 22:55:35.270966 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.270594    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.270966 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.270872    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgk8t\" (UniqueName: \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.271571 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.271511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.327610 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.327204    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgk8t\" (UniqueName: \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\") pod \"payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.390381 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.390339    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:35.777898 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:35.777812    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 22:55:35.782208 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:55:35.782173    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29557792_57fd_4908_8b2a_04dc5672b492.slice/crio-a08b506a895ec322636d6d9b1ed3061590b8e9e5f2f3ca334659dabdc67b7089 WatchSource:0}: Error finding container a08b506a895ec322636d6d9b1ed3061590b8e9e5f2f3ca334659dabdc67b7089: Status 404 returned error can't find the container with id a08b506a895ec322636d6d9b1ed3061590b8e9e5f2f3ca334659dabdc67b7089
Nov 01 22:55:36.506953 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:36.506914    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" event={"ID":"29557792-57fd-4908-8b2a-04dc5672b492","Type":"ContainerStarted","Data":"fadea0a8c34f7fd705d39980dea9521aa69ffb4ed00c24a4c11263d798c32b2a"}
Nov 01 22:55:36.507426 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:36.507003    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:55:36.507426 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:36.507024    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" event={"ID":"29557792-57fd-4908-8b2a-04dc5672b492","Type":"ContainerStarted","Data":"a08b506a895ec322636d6d9b1ed3061590b8e9e5f2f3ca334659dabdc67b7089"}
Nov 01 22:55:36.615099 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:36.615042    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" podStartSLOduration=1.615027354 podStartE2EDuration="1.615027354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:36.613135103 +0000 UTC m=+1890.569174947" watchObservedRunningTime="2026-11-01 22:55:36.615027354 +0000 UTC m=+1890.571067252"
Nov 01 22:55:40.627372 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:55:40.627330    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:55:47.518551 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:55:47.518453    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 22:56:10.628499 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:10.628457    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:56:39.141871 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.141825    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 22:56:39.149280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.149249    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.198399 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.198360    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f0206-serving-cert\""
Nov 01 22:56:39.202170 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.202141    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 22:56:39.250239 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.250195    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.250420 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.250309    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.250489 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.250469    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.250565 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.250533    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2h6c\" (UniqueName: \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.351961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.351915    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.352173 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:39.352063    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-serving-cert: secret "maas-api-e2e-inf-a-1f0206-serving-cert" not found
Nov 01 22:56:39.352173 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.352068    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.352173 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:39.352138    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls podName:579fa21b-e969-40a4-bff2-29b2f9579a12 nodeName:}" failed. No retries permitted until 2026-11-01 22:56:39.852120809 +0000 UTC m=+1953.808160659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls") pod "maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" (UID: "579fa21b-e969-40a4-bff2-29b2f9579a12") : secret "maas-api-e2e-inf-a-1f0206-serving-cert" not found
Nov 01 22:56:39.352374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.352210    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2h6c\" (UniqueName: \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.352374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.352364    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.352778 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.352755    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.355154 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.355134    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.418965 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.413772    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2h6c\" (UniqueName: \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.859902 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.859867    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:39.862799 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:39.862761    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:40.063879 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:40.063826    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:40.448526 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:40.448415    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 22:56:40.453214 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:56:40.453184    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579fa21b_e969_40a4_bff2_29b2f9579a12.slice/crio-7f97e417c7f05597e8cd7291b255cdb1ca74abd39a89b6f72ffa437b7ea1f027 WatchSource:0}: Error finding container 7f97e417c7f05597e8cd7291b255cdb1ca74abd39a89b6f72ffa437b7ea1f027: Status 404 returned error can't find the container with id 7f97e417c7f05597e8cd7291b255cdb1ca74abd39a89b6f72ffa437b7ea1f027
Nov 01 22:56:40.455090 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:40.455070    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:56:40.628291 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:40.628249    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:56:40.953744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:40.953710    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" event={"ID":"579fa21b-e969-40a4-bff2-29b2f9579a12","Type":"ContainerStarted","Data":"7f97e417c7f05597e8cd7291b255cdb1ca74abd39a89b6f72ffa437b7ea1f027"}
Nov 01 22:56:42.034091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:42.034042    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" event={"ID":"579fa21b-e969-40a4-bff2-29b2f9579a12","Type":"ContainerStarted","Data":"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"}
Nov 01 22:56:42.035257 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:42.035235    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:42.150457 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:42.150403    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" podStartSLOduration=2.34924352 podStartE2EDuration="3.150387872s" totalImagesPullingTime="801.144352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:42.146969923 +0000 UTC m=+1956.103009766" watchObservedRunningTime="2026-11-01 22:56:42.150387872 +0000 UTC m=+1956.106427716"
Nov 01 22:56:47.189001 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:47.188963    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:56:47.233207 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:47.233168    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:56:47.233478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:47.233441    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy" containerID="cri-o://574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c" gracePeriod=30
Nov 01 22:56:48.082765 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:48.082698    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" containerID="cri-o://8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd" gracePeriod=30
Nov 01 22:56:49.058331 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.058300    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 22:56:49.525927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.525838    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:56:49.525927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.525914    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:56:49.816644 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.816605    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 22:56:49.821576 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.821530    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.870084 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.870044    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d68bab-openshift-default-dockercfg-85xh4\""
Nov 01 22:56:49.926104 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.926072    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 22:56:49.948408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948369    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948408 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948412    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948627    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948692 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948679    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948792 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948774    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948833 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948803    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948877 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948832    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksgf\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.948944 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948930    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:49.949003 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:49.948987    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.050941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.050827    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.050941 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.050861    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.050993    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051028    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051093    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051118    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051327    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051336    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051361    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksgf\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051479 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051468    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051574    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051656    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.051939 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.051921    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.052067 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.052040    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.053829 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.053808    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.053935 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.053815    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.105002 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.104931    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.105348 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.105197    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksgf\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\") pod \"e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") " pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.136592 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.136514    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:50.459348 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.459258    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:56:50.459572 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.459460    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" containerID="cri-o://14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc" gracePeriod=30
Nov 01 22:56:50.507313 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.507276    5031 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:56:50.507498 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.507341    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:56:50.622442 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.622407    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 22:56:50.626485 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:56:50.626460    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf161fd_3ba6_4dab_baa4_be6e29b75c35.slice/crio-29afea8ab7d59fdb4a5b04ecc28d53c58b984da315b22147c41c5202e8de9e53 WatchSource:0}: Error finding container 29afea8ab7d59fdb4a5b04ecc28d53c58b984da315b22147c41c5202e8de9e53: Status 404 returned error can't find the container with id 29afea8ab7d59fdb4a5b04ecc28d53c58b984da315b22147c41c5202e8de9e53
Nov 01 22:56:50.629458 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.629417    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:56:50.629590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.629504    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:56:50.629590 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:50.629551    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:56:51.109681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:51.109643    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerStarted","Data":"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"}
Nov 01 22:56:51.110189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:51.109726    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerStarted","Data":"29afea8ab7d59fdb4a5b04ecc28d53c58b984da315b22147c41c5202e8de9e53"}
Nov 01 22:56:51.136825 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:51.136796    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:51.142642 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:51.142615    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:51.225826 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:51.225772    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podStartSLOduration=2.225753659 podStartE2EDuration="2.225753659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:51.222037049 +0000 UTC m=+1965.178076892" watchObservedRunningTime="2026-11-01 22:56:51.225753659 +0000 UTC m=+1965.181793504"
Nov 01 22:56:52.117198 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.117162    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:52.118288 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.118265    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:56:52.526328 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.526295    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:56:52.695447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695351    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695423    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695449    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695579    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695676    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695717    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695766 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695756    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695995 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695771    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.695995 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695845    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.695995 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.695992    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\") pod \"39489942-ba0d-45a9-90f9-5470817f18e4\" (UID: \"39489942-ba0d-45a9-90f9-5470817f18e4\") "
Nov 01 22:56:52.696167 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696064    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.696676 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696648    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.696858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696836    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.696933 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696880    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.696933 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696902    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.696933 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.696919    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39489942-ba0d-45a9-90f9-5470817f18e4-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.697099 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.697086    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.699674 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.699590    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.699774 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.699705    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "kube-api-access-9kcq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.701917 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.701882    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.702038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.702017    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token" pod "39489942-ba0d-45a9-90f9-5470817f18e4" (UID: "39489942-ba0d-45a9-90f9-5470817f18e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.798862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798822    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.798862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798857    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.798862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798868    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39489942-ba0d-45a9-90f9-5470817f18e4-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.799105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798877    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcq7\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-kube-api-access-9kcq7\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.799105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798888    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39489942-ba0d-45a9-90f9-5470817f18e4-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.799105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:52.798898    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39489942-ba0d-45a9-90f9-5470817f18e4-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.125586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.125531    5031 generic.go:356] "Generic (PLEG): container finished" podID="39489942-ba0d-45a9-90f9-5470817f18e4" containerID="574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c" exitCode=0
Nov 01 22:56:53.125586 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.125572    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" event={"ID":"39489942-ba0d-45a9-90f9-5470817f18e4","Type":"ContainerDied","Data":"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"}
Nov 01 22:56:53.126070 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.125598    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"
Nov 01 22:56:53.126070 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.125632    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj" event={"ID":"39489942-ba0d-45a9-90f9-5470817f18e4","Type":"ContainerDied","Data":"6d34309786a9cac5edbcb56ae5c9fe20954429c8f61a4c90022bb7030d93b37e"}
Nov 01 22:56:53.126070 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.125659    5031 scope.go:122] "RemoveContainer" containerID="574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"
Nov 01 22:56:53.197497 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.197456    5031 scope.go:122] "RemoveContainer" containerID="574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"
Nov 01 22:56:53.198206 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:53.198183    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c\": container with ID starting with 574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c not found: ID does not exist" containerID="574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"
Nov 01 22:56:53.198305 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.198216    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c"} err="failed to get container status \"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c\": rpc error: code = NotFound desc = could not find container \"574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c\": container with ID starting with 574cfe6512dde448e8a22226006b0b912f211fd6181864a4fd9e703a96cbb11c not found: ID does not exist"
Nov 01 22:56:53.280982 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.280940    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:56:53.326500 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.326458    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-6tbqj"]
Nov 01 22:56:53.388158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.388127    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:56:53.510987 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.510952    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511151    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511188    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511235    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511254    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.511327 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511281    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511487 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511468    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511587 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511522    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:53.511664 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511618    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511701    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511780 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511729    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.511780 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511754    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\") pod \"edee6deb-e0b4-49cc-89a4-a7ab88258786\" (UID: \"edee6deb-e0b4-49cc-89a4-a7ab88258786\") "
Nov 01 22:56:53.511944 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511909    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.512037 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.511955    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.512472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.512447    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.512603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.512482    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.512603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.512559    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.512603 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.512573    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.512807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.512675    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edee6deb-e0b4-49cc-89a4-a7ab88258786-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.514091 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.514059    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.514181 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.514115    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:53.514289 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.514271    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "kube-api-access-czqbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:53.514670 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.514651    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token" pod "edee6deb-e0b4-49cc-89a4-a7ab88258786" (UID: "edee6deb-e0b4-49cc-89a4-a7ab88258786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:53.618297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.615289    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.618297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.615328    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.618297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.615342    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqbb\" (UniqueName: \"kubernetes.io/projected/edee6deb-e0b4-49cc-89a4-a7ab88258786-kube-api-access-czqbb\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.618297 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:53.615359    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edee6deb-e0b4-49cc-89a4-a7ab88258786-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:54.134344 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.134309    5031 generic.go:356] "Generic (PLEG): container finished" podID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerID="8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd" exitCode=0
Nov 01 22:56:54.134858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.134358    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"
Nov 01 22:56:54.134858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.134392    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" event={"ID":"edee6deb-e0b4-49cc-89a4-a7ab88258786","Type":"ContainerDied","Data":"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"}
Nov 01 22:56:54.134858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.134443    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq" event={"ID":"edee6deb-e0b4-49cc-89a4-a7ab88258786","Type":"ContainerDied","Data":"2e6892651cd7eaab30e90f1778fd023260e8cff7da9c6d3e4eb9b53994d895f6"}
Nov 01 22:56:54.134858 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.134459    5031 scope.go:122] "RemoveContainer" containerID="8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"
Nov 01 22:56:54.148861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.148828    5031 scope.go:122] "RemoveContainer" containerID="8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"
Nov 01 22:56:54.149168 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:54.149147    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd\": container with ID starting with 8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd not found: ID does not exist" containerID="8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"
Nov 01 22:56:54.149226 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.149176    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd"} err="failed to get container status \"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd\": rpc error: code = NotFound desc = could not find container \"8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd\": container with ID starting with 8acb98de202d7a22cc4c261a5ccd9b36c55ed9d87553fd844356ba1d886fd7dd not found: ID does not exist"
Nov 01 22:56:54.174672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.174643    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" path="/var/lib/kubelet/pods/39489942-ba0d-45a9-90f9-5470817f18e4/volumes"
Nov 01 22:56:54.295691 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.295654    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:56:54.341635 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:54.341601    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-qlmbq"]
Nov 01 22:56:55.748776 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.748745    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:56:55.847471 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.847372    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.847738 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.847713    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.847872 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.847794    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848000 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.847880    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848078 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848026    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848081    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848113    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848304 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848142    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848304 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848269    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\") pod \"53be7b1c-747d-4375-b79e-1550be77f7d5\" (UID: \"53be7b1c-747d-4375-b79e-1550be77f7d5\") "
Nov 01 22:56:55.848675 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848634    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.849158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848832    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.849158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848854    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.849158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.848780    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.849158 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849103    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:55.849428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849306    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.849428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849328    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.849428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849344    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.849428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849359    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53be7b1c-747d-4375-b79e-1550be77f7d5-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.849428 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.849372    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.850352 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.850329    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:55.850672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.850645    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:55.850782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.850692    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.851041 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.851017    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g" pod "53be7b1c-747d-4375-b79e-1550be77f7d5" (UID: "53be7b1c-747d-4375-b79e-1550be77f7d5"). InnerVolumeSpecName "kube-api-access-dh42g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:55.952189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.952153    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.952189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.952187    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.952189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.952197    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh42g\" (UniqueName: \"kubernetes.io/projected/53be7b1c-747d-4375-b79e-1550be77f7d5-kube-api-access-dh42g\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.952471 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:55.952209    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53be7b1c-747d-4375-b79e-1550be77f7d5-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.154188 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.154151    5031 generic.go:356] "Generic (PLEG): container finished" podID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerID="14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc" exitCode=0
Nov 01 22:56:56.154355 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.154256    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"
Nov 01 22:56:56.154355 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.154254    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" event={"ID":"53be7b1c-747d-4375-b79e-1550be77f7d5","Type":"ContainerDied","Data":"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"}
Nov 01 22:56:56.154476 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.154391    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c" event={"ID":"53be7b1c-747d-4375-b79e-1550be77f7d5","Type":"ContainerDied","Data":"036468aa808ed6b271ee61f41c055d9b60446e8187be8e8890799fb062e3a157"}
Nov 01 22:56:56.154476 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.154422    5031 scope.go:122] "RemoveContainer" containerID="14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"
Nov 01 22:56:56.166454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.166427    5031 scope.go:122] "RemoveContainer" containerID="14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"
Nov 01 22:56:56.166837 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:56:56.166813    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc\": container with ID starting with 14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc not found: ID does not exist" containerID="14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"
Nov 01 22:56:56.166889 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.166846    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc"} err="failed to get container status \"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc\": rpc error: code = NotFound desc = could not find container \"14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc\": container with ID starting with 14e48eabd58ed6948d7f3ad2bd08b8e00c512091b323fa96b85fed93d0115dcc not found: ID does not exist"
Nov 01 22:56:56.173523 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.173492    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" path="/var/lib/kubelet/pods/edee6deb-e0b4-49cc-89a4-a7ab88258786/volumes"
Nov 01 22:56:56.312937 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.312902    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:56:56.366167 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:56.366133    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-lst6c"]
Nov 01 22:56:58.180961 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:56:58.180930    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" path="/var/lib/kubelet/pods/53be7b1c-747d-4375-b79e-1550be77f7d5/volumes"
Nov 01 22:57:10.632231 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:57:10.632186    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:57:40.629004 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:57:40.628911    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:58:05.854225 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:05.854189    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:58:05.855293 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:05.854415    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" containerID="cri-o://736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85" gracePeriod=30
Nov 01 22:58:07.923118 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:07.923068    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:58:07.923506 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:07.923152    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:58:08.156719 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.156674    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 22:58:08.157445 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157422    5031 cpu_manager.go:414] "containerMap: removing container" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy"
Nov 01 22:58:08.157605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157447    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy"
Nov 01 22:58:08.157605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157498    5031 cpu_manager.go:414] "containerMap: removing container" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy"
Nov 01 22:58:08.157605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157506    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy"
Nov 01 22:58:08.157605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157561    5031 cpu_manager.go:414] "containerMap: removing container" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy"
Nov 01 22:58:08.157605 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157569    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy"
Nov 01 22:58:08.157861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157789    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="39489942-ba0d-45a9-90f9-5470817f18e4" containerName="istio-proxy"
Nov 01 22:58:08.157861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157813    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="53be7b1c-747d-4375-b79e-1550be77f7d5" containerName="istio-proxy"
Nov 01 22:58:08.157861 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.157829    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="edee6deb-e0b4-49cc-89a4-a7ab88258786" containerName="istio-proxy"
Nov 01 22:58:08.161613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.161591    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305200 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305160    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305222    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305456 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305410    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305649 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305486    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305649 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305532    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcx9g\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305649 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305610    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305656    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305782 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305765    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.305863 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.305827    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.315748 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.315723    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-945efd02-openshift-default-dockercfg-fl45l\""
Nov 01 22:58:08.366330 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.366292    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 22:58:08.407616 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.407582    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.407895 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.407871    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcx9g\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408014 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.407994    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408105 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408089    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408205    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408436 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408400    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408664 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408637    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408808 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408726    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408808 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408745    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408809    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.408946 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.408910    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.409052 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.409021    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.409267 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.409245    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.409840 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.409800    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.411696 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.411671    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.412310 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.412286    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.461757 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.461724    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.461932 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.461876    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcx9g\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.476773 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.476739    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:08.905510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.905465    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 22:58:08.910096 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:58:08.910061    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3886168_b1b1_42c6_9bd6_3d4e5b631253.slice/crio-0a5ed702bb9564845657447838ac402e97cafae38d841f9d34b5b76aad09d577 WatchSource:0}: Error finding container 0a5ed702bb9564845657447838ac402e97cafae38d841f9d34b5b76aad09d577: Status 404 returned error can't find the container with id 0a5ed702bb9564845657447838ac402e97cafae38d841f9d34b5b76aad09d577
Nov 01 22:58:08.913027 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.912991    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:08.913100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.913088    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:08.913139 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:08.913123    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:09.685749 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:09.685708    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" event={"ID":"b3886168-b1b1-42c6-9bd6-3d4e5b631253","Type":"ContainerStarted","Data":"50e5b04830385eff9e0b92cd2cc1a37cf329a402770c2f44006f2d4a73343b2d"}
Nov 01 22:58:09.686164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:09.685769    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" event={"ID":"b3886168-b1b1-42c6-9bd6-3d4e5b631253","Type":"ContainerStarted","Data":"0a5ed702bb9564845657447838ac402e97cafae38d841f9d34b5b76aad09d577"}
Nov 01 22:58:09.847514 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:09.847452    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podStartSLOduration=1.8474339130000001 podStartE2EDuration="1.847433913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:09.844061586 +0000 UTC m=+2043.800101435" watchObservedRunningTime="2026-11-01 22:58:09.847433913 +0000 UTC m=+2043.803473756"
Nov 01 22:58:10.478817 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:10.478769    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:10.480785 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:10.480757    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:10.480934 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:10.480822    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:10.627619 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:10.627577    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:58:11.217607 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.217583    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:58:11.251678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251642    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.251868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251689    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.251868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251744    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.251868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251781    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.251868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251809    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.251868 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251866    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.252141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251945    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.252141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.251972    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.252141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252002    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:58:11.252141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252027    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\") pod \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\" (UID: \"df8a8640-4e1f-40da-9a4a-50be556aa5b6\") "
Nov 01 22:58:11.252141 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252090    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:58:11.252400 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252328    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:58:11.252400 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252386    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:11.252588 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.252533    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:58:11.253276 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.253253    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.253370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.253292    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.253370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.253314    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.253370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.253328    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.253370 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.253342    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.255043 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.255000    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:11.255296 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.255267    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "kube-api-access-69hb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:11.255375 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.255337    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:58:11.255472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.255449    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo" pod "df8a8640-4e1f-40da-9a4a-50be556aa5b6" (UID: "df8a8640-4e1f-40da-9a4a-50be556aa5b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:58:11.355100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.354982    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.355100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.355026    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hb9\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-kube-api-access-69hb9\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.355100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.355043    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.355100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.355056    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8a8640-4e1f-40da-9a4a-50be556aa5b6-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:11.479312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.479053    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:11.479312 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.479116    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:11.703057 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.703019    5031 generic.go:356] "Generic (PLEG): container finished" podID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerID="736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85" exitCode=0
Nov 01 22:58:11.703254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.703075    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"
Nov 01 22:58:11.703254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.703074    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" event={"ID":"df8a8640-4e1f-40da-9a4a-50be556aa5b6","Type":"ContainerDied","Data":"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"}
Nov 01 22:58:11.703254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.703140    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn" event={"ID":"df8a8640-4e1f-40da-9a4a-50be556aa5b6","Type":"ContainerDied","Data":"a7d92e4094b04ff468ecfccbc7b155479ae0420edea8f462677ef5f4cfc63630"}
Nov 01 22:58:11.703254 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.703165    5031 scope.go:122] "RemoveContainer" containerID="736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"
Nov 01 22:58:11.715366 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.715346    5031 scope.go:122] "RemoveContainer" containerID="736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"
Nov 01 22:58:11.715716 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:11.715694    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85\": container with ID starting with 736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85 not found: ID does not exist" containerID="736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"
Nov 01 22:58:11.715780 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.715726    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85"} err="failed to get container status \"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85\": rpc error: code = NotFound desc = could not find container \"736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85\": container with ID starting with 736b15a423e4405c97e894f8a2d4be0f566535a900ef2bb49c1c85157cd1df85 not found: ID does not exist"
Nov 01 22:58:11.883307 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.883268    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:58:11.886630 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:11.886602    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-gkqbn"]
Nov 01 22:58:12.186427 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:12.186342    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" path="/var/lib/kubelet/pods/df8a8640-4e1f-40da-9a4a-50be556aa5b6/volumes"
Nov 01 22:58:12.478681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:12.478035    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:12.478681 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:12.478105    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:13.479038 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:13.478997    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:13.479453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:13.479090    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:14.479236 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:14.479194    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:14.479793 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:14.479301    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:15.477862 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:15.477821    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:58:15.478062 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:15.477896    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 01 22:58:16.482927 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:16.482899    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:16.483422 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:16.483400    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:16.484133 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:16.484112    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 22:58:16.580832 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:16.580794    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:58:16.662815 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:16.662784    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:58:21.581751 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:21.581660    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" podUID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" containerName="payload-processing" containerID="cri-o://ae6ce2580a131e66a4ceda691343907c6599c72f24d8a06d082407bb27617645" gracePeriod=25
Nov 01 22:58:21.663685 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:21.663623    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" podUID="f8de0329-3d28-42f1-97c4-e88b85452011" containerName="payload-pre-processing" containerID="cri-o://666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d" gracePeriod=25
Nov 01 22:58:21.774645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:21.774611    5031 generic.go:356] "Generic (PLEG): container finished" podID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" containerID="ae6ce2580a131e66a4ceda691343907c6599c72f24d8a06d082407bb27617645" exitCode=0
Nov 01 22:58:21.774813 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:21.774684    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" event={"ID":"0b8ce478-dcb8-4fec-9db3-318ca987fc2a","Type":"ContainerDied","Data":"ae6ce2580a131e66a4ceda691343907c6599c72f24d8a06d082407bb27617645"}
Nov 01 22:58:21.975670 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:21.975626    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:58:22.011848 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.011824    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:58:22.037613 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.037581    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\" (UniqueName: \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\") pod \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") "
Nov 01 22:58:22.037781 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.037667    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\") pod \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\" (UID: \"0b8ce478-dcb8-4fec-9db3-318ca987fc2a\") "
Nov 01 22:58:22.038102 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.038081    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume" pod "0b8ce478-dcb8-4fec-9db3-318ca987fc2a" (UID: "0b8ce478-dcb8-4fec-9db3-318ca987fc2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:22.038221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.038207    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:22.039879 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.039862    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5" pod "0b8ce478-dcb8-4fec-9db3-318ca987fc2a" (UID: "0b8ce478-dcb8-4fec-9db3-318ca987fc2a"). InnerVolumeSpecName "kube-api-access-4vts5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:22.139483 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.139393    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\" (UniqueName: \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\") pod \"f8de0329-3d28-42f1-97c4-e88b85452011\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") "
Nov 01 22:58:22.139659 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.139601    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\") pod \"f8de0329-3d28-42f1-97c4-e88b85452011\" (UID: \"f8de0329-3d28-42f1-97c4-e88b85452011\") "
Nov 01 22:58:22.139964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.139937    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume" pod "f8de0329-3d28-42f1-97c4-e88b85452011" (UID: "f8de0329-3d28-42f1-97c4-e88b85452011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:22.140299 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.140261    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vts5\" (UniqueName: \"kubernetes.io/projected/0b8ce478-dcb8-4fec-9db3-318ca987fc2a-kube-api-access-4vts5\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:22.140357 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.140309    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8de0329-3d28-42f1-97c4-e88b85452011-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:22.141818 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.141796    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8" pod "f8de0329-3d28-42f1-97c4-e88b85452011" (UID: "f8de0329-3d28-42f1-97c4-e88b85452011"). InnerVolumeSpecName "kube-api-access-tprf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:22.242060 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.242015    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tprf8\" (UniqueName: \"kubernetes.io/projected/f8de0329-3d28-42f1-97c4-e88b85452011-kube-api-access-tprf8\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:22.783472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.783442    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"
Nov 01 22:58:22.783472 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.783458    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd" event={"ID":"0b8ce478-dcb8-4fec-9db3-318ca987fc2a","Type":"ContainerDied","Data":"38e3ff294748f6d05cf351c6228fc05b2ea591fff5f17a449156f89511a965c1"}
Nov 01 22:58:22.783994 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.783531    5031 scope.go:122] "RemoveContainer" containerID="ae6ce2580a131e66a4ceda691343907c6599c72f24d8a06d082407bb27617645"
Nov 01 22:58:22.785347 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.785320    5031 generic.go:356] "Generic (PLEG): container finished" podID="f8de0329-3d28-42f1-97c4-e88b85452011" containerID="666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d" exitCode=0
Nov 01 22:58:22.785453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.785358    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" event={"ID":"f8de0329-3d28-42f1-97c4-e88b85452011","Type":"ContainerDied","Data":"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"}
Nov 01 22:58:22.785453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.785379    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"
Nov 01 22:58:22.785453 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.785423    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq" event={"ID":"f8de0329-3d28-42f1-97c4-e88b85452011","Type":"ContainerDied","Data":"8d465fd07491670d006f8b4bc53853ec25ec5d2110d51e2c72afa52fc3669d23"}
Nov 01 22:58:22.794698 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.794670    5031 scope.go:122] "RemoveContainer" containerID="666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"
Nov 01 22:58:22.804374 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.804352    5031 scope.go:122] "RemoveContainer" containerID="666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"
Nov 01 22:58:22.804740 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:22.804719    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d\": container with ID starting with 666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d not found: ID does not exist" containerID="666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"
Nov 01 22:58:22.804819 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.804748    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d"} err="failed to get container status \"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d\": rpc error: code = NotFound desc = could not find container \"666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d\": container with ID starting with 666bb404a39a78432d541156ff5c30e1ec5dc90591fbfe7df633d641e8bd212d not found: ID does not exist"
Nov 01 22:58:22.937423 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.937339    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:58:22.983556 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:22.983507    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bd7ff8-56b48755c-mfnzd"]
Nov 01 22:58:23.043748 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.043708    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:58:23.047363 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.047334    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bd7ff8-794bc58b95-gmhwq"]
Nov 01 22:58:23.269573 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.269469    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 22:58:23.270138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270118    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f8de0329-3d28-42f1-97c4-e88b85452011" containerName="payload-pre-processing"
Nov 01 22:58:23.270138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270136    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8de0329-3d28-42f1-97c4-e88b85452011" containerName="payload-pre-processing"
Nov 01 22:58:23.270298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270157    5031 cpu_manager.go:414] "containerMap: removing container" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy"
Nov 01 22:58:23.270298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270165    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy"
Nov 01 22:58:23.270298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270200    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" containerName="payload-processing"
Nov 01 22:58:23.270298 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270206    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" containerName="payload-processing"
Nov 01 22:58:23.270424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270313    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" containerName="payload-processing"
Nov 01 22:58:23.270424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270325    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8de0329-3d28-42f1-97c4-e88b85452011" containerName="payload-pre-processing"
Nov 01 22:58:23.270424 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.270332    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8a8640-4e1f-40da-9a4a-50be556aa5b6" containerName="istio-proxy"
Nov 01 22:58:23.275759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.275736    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.285258 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.285231    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 22:58:23.289700 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.289678    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.333273 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.333236    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 22:58:23.357138 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357094    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357153    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxf7\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357243    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357336 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357302    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357390    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357512 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357473    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357645 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357586    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357663    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.357759 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.357715    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.386407 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.386372    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 22:58:23.459054 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459018    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459081    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459260 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459168    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459378 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459261    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459378 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459283    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459378 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459307    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459378 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459328    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459385    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459443    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459590    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459660 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459624    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459924 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459742    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.459924 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459776    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459924 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459870    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.459924 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.459897    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460125 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460002    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvpz\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.460125 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460038    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460122    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxf7\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460167    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460402 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460233    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460589 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460514    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.460651 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.460584    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.462063 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.462040    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.462291 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.462273    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.512454 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.512424    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.516791 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.516757    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxf7\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.561816 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.561779    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.561831    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.561892    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.561929    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.561957    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562017    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562085    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562162    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvpz\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562286 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562236    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562284    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562332    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562441 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562387    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562895 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.562988 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.562944    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.564425 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.564405    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.564736 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.564719    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.592393 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.592353    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:23.615442 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.615410    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.615720 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.615696    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvpz\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:23.904717 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:23.904627    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:24.026052 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.026012    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 22:58:24.030643 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:58:24.030611    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca60c084_efcc_4e56_a268_1cc20fce1c40.slice/crio-c7ba06b46171aa91491f471064b960f9bfa6326c7334a9a4a33e3be352ff3fde WatchSource:0}: Error finding container c7ba06b46171aa91491f471064b960f9bfa6326c7334a9a4a33e3be352ff3fde: Status 404 returned error can't find the container with id c7ba06b46171aa91491f471064b960f9bfa6326c7334a9a4a33e3be352ff3fde
Nov 01 22:58:24.044758 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.044687    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.044943 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.044864    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.045013 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.044959    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.176124 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.176040    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8ce478-dcb8-4fec-9db3-318ca987fc2a" path="/var/lib/kubelet/pods/0b8ce478-dcb8-4fec-9db3-318ca987fc2a/volumes"
Nov 01 22:58:24.176678 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.176660    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8de0329-3d28-42f1-97c4-e88b85452011" path="/var/lib/kubelet/pods/f8de0329-3d28-42f1-97c4-e88b85452011/volumes"
Nov 01 22:58:24.302373 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.302322    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 22:58:24.313478 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:58:24.313439    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb503ae4b_0ac0_434f_a8e3_2b728724ef3a.slice/crio-15cc14fb8258ca9284aed03ef07b93605438de0f57c2c985eb0dd0525bbb257f WatchSource:0}: Error finding container 15cc14fb8258ca9284aed03ef07b93605438de0f57c2c985eb0dd0525bbb257f: Status 404 returned error can't find the container with id 15cc14fb8258ca9284aed03ef07b93605438de0f57c2c985eb0dd0525bbb257f
Nov 01 22:58:24.317148 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.317098    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.317301 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.317193    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.317301 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.317242    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:58:24.816569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.816234    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" event={"ID":"ca60c084-efcc-4e56-a268-1cc20fce1c40","Type":"ContainerStarted","Data":"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"}
Nov 01 22:58:24.816569 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.816362    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" event={"ID":"ca60c084-efcc-4e56-a268-1cc20fce1c40","Type":"ContainerStarted","Data":"c7ba06b46171aa91491f471064b960f9bfa6326c7334a9a4a33e3be352ff3fde"}
Nov 01 22:58:24.822079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.819491    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" event={"ID":"b503ae4b-0ac0-434f-a8e3-2b728724ef3a","Type":"ContainerStarted","Data":"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"}
Nov 01 22:58:24.822079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.819616    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" event={"ID":"b503ae4b-0ac0-434f-a8e3-2b728724ef3a","Type":"ContainerStarted","Data":"15cc14fb8258ca9284aed03ef07b93605438de0f57c2c985eb0dd0525bbb257f"}
Nov 01 22:58:24.905389 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.905357    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:24.933887 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:24.933825    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podStartSLOduration=1.933810923 podStartE2EDuration="1.933810923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:24.931071391 +0000 UTC m=+2058.887111256" watchObservedRunningTime="2026-11-01 22:58:24.933810923 +0000 UTC m=+2058.889850764"
Nov 01 22:58:25.044157 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.044100    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podStartSLOduration=2.044086253 podStartE2EDuration="2.044086253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:25.041617615 +0000 UTC m=+2058.997657460" watchObservedRunningTime="2026-11-01 22:58:25.044086253 +0000 UTC m=+2059.000126101"
Nov 01 22:58:25.594563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.594279    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:25.594563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.594421    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:25.594563 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.594464    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:25.914377 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.914280    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:25.915004 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:25.914968    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:26.595648 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:26.595033    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:26.595648 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:26.595099    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:26.908080 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:26.907987    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:26.908080 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:26.908044    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:27.594562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:27.594139    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:27.594562 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:27.594218    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:27.915510 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:27.909047    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:27.915903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:27.915852    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:28.599566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:28.594009    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:28.599566 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:28.594081    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:28.906081 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:28.905983    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:28.906337 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:28.906309    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:29.602744 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:29.602706    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:29.603279 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:29.602790    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:29.906034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:29.905846    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:29.906034 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:29.905913    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:30.593498 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:30.593458    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:58:30.593718 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:30.593523    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 01 22:58:30.906183 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:30.906083    5031 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:58:30.906183 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:30.906146    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 22:58:31.598469 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.598438    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:31.598949 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.598931    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:31.599796 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.599779    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 22:58:31.912828 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.912745    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:31.913290 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.913261    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:31.914262 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:31.914241    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 22:58:40.627509 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:40.627467    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:58:42.524812 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:42.524775    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:58:42.638754 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:42.638706    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:58:47.526079 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:47.525993    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" podUID="d1249469-a0c5-42e6-88a1-d88edec812e2" containerName="payload-processing" containerID="cri-o://1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861" gracePeriod=25
Nov 01 22:58:47.640434 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:47.640351    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" podUID="8e96df97-353f-4988-aba1-d95289917201" containerName="payload-pre-processing" containerID="cri-o://954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7" gracePeriod=25
Nov 01 22:58:47.851807 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:47.851771    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:58:47.940148 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:47.940119    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:58:48.018162 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.018128    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\" (UniqueName: \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\") pod \"d1249469-a0c5-42e6-88a1-d88edec812e2\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") "
Nov 01 22:58:48.018332 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.018171    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\") pod \"d1249469-a0c5-42e6-88a1-d88edec812e2\" (UID: \"d1249469-a0c5-42e6-88a1-d88edec812e2\") "
Nov 01 22:58:48.018672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.018642    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume" pod "d1249469-a0c5-42e6-88a1-d88edec812e2" (UID: "d1249469-a0c5-42e6-88a1-d88edec812e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:48.018952 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.018934    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1249469-a0c5-42e6-88a1-d88edec812e2-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:48.020447 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.020420    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2" pod "d1249469-a0c5-42e6-88a1-d88edec812e2" (UID: "d1249469-a0c5-42e6-88a1-d88edec812e2"). InnerVolumeSpecName "kube-api-access-nzcl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:48.109231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.109131    5031 generic.go:356] "Generic (PLEG): container finished" podID="8e96df97-353f-4988-aba1-d95289917201" containerID="954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7" exitCode=0
Nov 01 22:58:48.109231 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.109182    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"
Nov 01 22:58:48.109478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.109219    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" event={"ID":"8e96df97-353f-4988-aba1-d95289917201","Type":"ContainerDied","Data":"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"}
Nov 01 22:58:48.109478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.109291    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd" event={"ID":"8e96df97-353f-4988-aba1-d95289917201","Type":"ContainerDied","Data":"ebd335d047d0090ef74f66f992b37be6eb042bc5561e3f68a7160f56905bd591"}
Nov 01 22:58:48.109478 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.109320    5031 scope.go:122] "RemoveContainer" containerID="954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"
Nov 01 22:58:48.111222 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.111192    5031 generic.go:356] "Generic (PLEG): container finished" podID="d1249469-a0c5-42e6-88a1-d88edec812e2" containerID="1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861" exitCode=0
Nov 01 22:58:48.111433 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.111229    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" event={"ID":"d1249469-a0c5-42e6-88a1-d88edec812e2","Type":"ContainerDied","Data":"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"}
Nov 01 22:58:48.111433 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.111255    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"
Nov 01 22:58:48.111433 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.111282    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8" event={"ID":"d1249469-a0c5-42e6-88a1-d88edec812e2","Type":"ContainerDied","Data":"4fa6671c4c1ef2359e8b8c254598a9a685f7d75c4c1a20e601c0e3c4ed11da41"}
Nov 01 22:58:48.120398 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.120365    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\") pod \"8e96df97-353f-4988-aba1-d95289917201\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") "
Nov 01 22:58:48.121672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.120715    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\" (UniqueName: \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\") pod \"8e96df97-353f-4988-aba1-d95289917201\" (UID: \"8e96df97-353f-4988-aba1-d95289917201\") "
Nov 01 22:58:48.121672 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.121318    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcl2\" (UniqueName: \"kubernetes.io/projected/d1249469-a0c5-42e6-88a1-d88edec812e2-kube-api-access-nzcl2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:48.121964 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.121937    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume" pod "8e96df97-353f-4988-aba1-d95289917201" (UID: "8e96df97-353f-4988-aba1-d95289917201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:48.122323 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.122298    5031 scope.go:122] "RemoveContainer" containerID="954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"
Nov 01 22:58:48.122737 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:48.122706    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7\": container with ID starting with 954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7 not found: ID does not exist" containerID="954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"
Nov 01 22:58:48.122823 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.122746    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7"} err="failed to get container status \"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7\": rpc error: code = NotFound desc = could not find container \"954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7\": container with ID starting with 954142e90a898b4163f4772cda22ff1817bed4ac9d99c82d690c75620a6e49d7 not found: ID does not exist"
Nov 01 22:58:48.122823 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.122772    5031 scope.go:122] "RemoveContainer" containerID="1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"
Nov 01 22:58:48.123904 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.123867    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5" pod "8e96df97-353f-4988-aba1-d95289917201" (UID: "8e96df97-353f-4988-aba1-d95289917201"). InnerVolumeSpecName "kube-api-access-pvkf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:48.134432 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.134403    5031 scope.go:122] "RemoveContainer" containerID="1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"
Nov 01 22:58:48.135299 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:48.135273    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861\": container with ID starting with 1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861 not found: ID does not exist" containerID="1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"
Nov 01 22:58:48.135385 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.135310    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861"} err="failed to get container status \"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861\": rpc error: code = NotFound desc = could not find container \"1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861\": container with ID starting with 1a166a39185b14867fbee71a9ed2552bc492ab887978460b553f9962c8219861 not found: ID does not exist"
Nov 01 22:58:48.223097 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.223059    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e96df97-353f-4988-aba1-d95289917201-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:48.223097 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.223091    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvkf5\" (UniqueName: \"kubernetes.io/projected/8e96df97-353f-4988-aba1-d95289917201-kube-api-access-pvkf5\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:48.262960 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.262916    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:58:48.268683 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.268653    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f5b935-58d6cfc997-n78c8"]
Nov 01 22:58:48.516804 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.516708    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:58:48.520368 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:48.520331    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-f5b935-5df5f87c9f-bm4sd"]
Nov 01 22:58:50.172204 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.172157    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e96df97-353f-4988-aba1-d95289917201" path="/var/lib/kubelet/pods/8e96df97-353f-4988-aba1-d95289917201/volumes"
Nov 01 22:58:50.172654 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.172640    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1249469-a0c5-42e6-88a1-d88edec812e2" path="/var/lib/kubelet/pods/d1249469-a0c5-42e6-88a1-d88edec812e2/volumes"
Nov 01 22:58:50.421280 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.421244    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 22:58:50.422026 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422005    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8e96df97-353f-4988-aba1-d95289917201" containerName="payload-pre-processing"
Nov 01 22:58:50.422100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422030    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e96df97-353f-4988-aba1-d95289917201" containerName="payload-pre-processing"
Nov 01 22:58:50.422100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422042    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d1249469-a0c5-42e6-88a1-d88edec812e2" containerName="payload-processing"
Nov 01 22:58:50.422100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422049    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1249469-a0c5-42e6-88a1-d88edec812e2" containerName="payload-processing"
Nov 01 22:58:50.422194 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422172    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e96df97-353f-4988-aba1-d95289917201" containerName="payload-pre-processing"
Nov 01 22:58:50.422194 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.422182    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1249469-a0c5-42e6-88a1-d88edec812e2" containerName="payload-processing"
Nov 01 22:58:50.427922 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.427854    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.455431 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.455390    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.455663 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.455563    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.455742 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.455701    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mq2\" (UniqueName: \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.455840 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.455823    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.479087 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.479053    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-71505e-serving-cert\""
Nov 01 22:58:50.531072 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.531012    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 22:58:50.557708 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.557661    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.557903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.557781    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.557975 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:50.557891    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-serving-cert: secret "maas-api-e2e-api-a-71505e-serving-cert" not found
Nov 01 22:58:50.557975 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.557932    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mq2\" (UniqueName: \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.557975 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:58:50.557972    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls podName:5f9910a1-b517-4572-9a16-17da3b92450b nodeName:}" failed. No retries permitted until 2026-11-01 22:58:51.057950195 +0000 UTC m=+2085.013990021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls") pod "maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" (UID: "5f9910a1-b517-4572-9a16-17da3b92450b") : secret "maas-api-e2e-api-a-71505e-serving-cert" not found
Nov 01 22:58:50.558189 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.558116    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.559050 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.559025    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.560501 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.560477    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:50.612247 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:50.612216    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mq2\" (UniqueName: \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:51.065205 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:51.065167    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:51.068215 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:51.068188    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") pod \"maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:51.341903 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:51.341805    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:51.726740 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:51.726651    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 22:58:51.732112 ip-10-0-138-155 kubenswrapper[5031]: W1101 22:58:51.732075    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9910a1_b517_4572_9a16_17da3b92450b.slice/crio-4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c WatchSource:0}: Error finding container 4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c: Status 404 returned error can't find the container with id 4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c
Nov 01 22:58:52.143364 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:52.143333    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" event={"ID":"5f9910a1-b517-4572-9a16-17da3b92450b","Type":"ContainerStarted","Data":"4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c"}
Nov 01 22:58:53.151251 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:53.151211    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" event={"ID":"5f9910a1-b517-4572-9a16-17da3b92450b","Type":"ContainerStarted","Data":"9f82af32a6385931d7f1e7c4e955e1fe94464ec7e2f999be6e312f623d7040b1"}
Nov 01 22:58:53.151737 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:53.151317    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:58:53.265388 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:53.265319    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" podStartSLOduration=2.406405787 podStartE2EDuration="3.265301964s" totalImagesPullingTime="858.896177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:53.262242782 +0000 UTC m=+2087.218282627" watchObservedRunningTime="2026-11-01 22:58:53.265301964 +0000 UTC m=+2087.221341807"
Nov 01 22:58:59.164864 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:58:59.164833    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 22:59:10.628330 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:59:10.628290    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 22:59:17.349702 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.349611    5031 generic.go:356] "Generic (PLEG): container finished" podID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerID="96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6" exitCode=0
Nov 01 22:59:17.350145 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.349713    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerDied","Data":"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"}
Nov 01 22:59:17.350221 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.350201    5031 scope.go:122] "RemoveContainer" containerID="96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"
Nov 01 22:59:17.352164 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.352135    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:59:17.352283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.352214    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:59:17.352283 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:17.352248    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 22:59:18.390119 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:18.390060    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerStarted","Data":"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"}
Nov 01 22:59:19.137202 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:19.137163    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:59:19.138449 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:19.138420    5031 patch_prober.go:29] interesting pod/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:59:19.138581 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:19.138470    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 22:59:20.137009 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:20.136971    5031 patch_prober.go:29] interesting pod/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:59:20.137488 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:20.137033    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 22:59:21.137233 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:21.137197    5031 patch_prober.go:29] interesting pod/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:59:21.137674 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:21.137249    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 22:59:22.118201 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:22.118148    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:59:22.137100 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:22.137067    5031 patch_prober.go:29] interesting pod/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:59:22.137274 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:22.137135    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 22:59:23.137207 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:23.137167    5031 patch_prober.go:29] interesting pod/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 01 22:59:23.137410 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:23.137236    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 01 22:59:24.143170 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:24.143136    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:59:24.144409 ip-10-0-138-155 kubenswrapper[5031]: I1101 22:59:24.144387    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 22:59:40.627297 ip-10-0-138-155 kubenswrapper[5031]: E1101 22:59:40.627257    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:00:05.961635 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:05.961595    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 23:00:05.962292 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:05.961856    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" podUID="579fa21b-e969-40a4-bff2-29b2f9579a12" containerName="maas-api" containerID="cri-o://89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef" gracePeriod=30
Nov 01 23:00:06.341645 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.341609    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 23:00:06.467724 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.467691    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\" (UniqueName: \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\") pod \"579fa21b-e969-40a4-bff2-29b2f9579a12\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") "
Nov 01 23:00:06.467916 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.467824    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\") pod \"579fa21b-e969-40a4-bff2-29b2f9579a12\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") "
Nov 01 23:00:06.467916 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.467882    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") pod \"579fa21b-e969-40a4-bff2-29b2f9579a12\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") "
Nov 01 23:00:06.467916 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.467899    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\") pod \"579fa21b-e969-40a4-bff2-29b2f9579a12\" (UID: \"579fa21b-e969-40a4-bff2-29b2f9579a12\") "
Nov 01 23:00:06.468281 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.468253    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca" pod "579fa21b-e969-40a4-bff2-29b2f9579a12" (UID: "579fa21b-e969-40a4-bff2-29b2f9579a12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:06.468509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.468487    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/579fa21b-e969-40a4-bff2-29b2f9579a12-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:06.470235 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.470210    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c" pod "579fa21b-e969-40a4-bff2-29b2f9579a12" (UID: "579fa21b-e969-40a4-bff2-29b2f9579a12"). InnerVolumeSpecName "kube-api-access-k2h6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:06.470450 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.470430    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs" pod "579fa21b-e969-40a4-bff2-29b2f9579a12" (UID: "579fa21b-e969-40a4-bff2-29b2f9579a12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:06.470527 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.470464    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls" pod "579fa21b-e969-40a4-bff2-29b2f9579a12" (UID: "579fa21b-e969-40a4-bff2-29b2f9579a12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:06.570308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.570226    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:06.570308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.570258    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/579fa21b-e969-40a4-bff2-29b2f9579a12-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:06.570308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.570270    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2h6c\" (UniqueName: \"kubernetes.io/projected/579fa21b-e969-40a4-bff2-29b2f9579a12-kube-api-access-k2h6c\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:06.745447 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.745414    5031 generic.go:356] "Generic (PLEG): container finished" podID="579fa21b-e969-40a4-bff2-29b2f9579a12" containerID="89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef" exitCode=0
Nov 01 23:00:06.745677 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.745466    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"
Nov 01 23:00:06.745677 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.745506    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" event={"ID":"579fa21b-e969-40a4-bff2-29b2f9579a12","Type":"ContainerDied","Data":"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"}
Nov 01 23:00:06.745677 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.745602    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn" event={"ID":"579fa21b-e969-40a4-bff2-29b2f9579a12","Type":"ContainerDied","Data":"7f97e417c7f05597e8cd7291b255cdb1ca74abd39a89b6f72ffa437b7ea1f027"}
Nov 01 23:00:06.745677 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.745643    5031 scope.go:122] "RemoveContainer" containerID="89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"
Nov 01 23:00:06.756630 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.756607    5031 scope.go:122] "RemoveContainer" containerID="89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"
Nov 01 23:00:06.756968 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:06.756947    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef\": container with ID starting with 89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef not found: ID does not exist" containerID="89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"
Nov 01 23:00:06.757019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.756977    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef"} err="failed to get container status \"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef\": rpc error: code = NotFound desc = could not find container \"89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef\": container with ID starting with 89cdfed3807440491fef737644fccc5a79545a4571aaa7c1d0f2e225370513ef not found: ID does not exist"
Nov 01 23:00:06.921202 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.920268    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 23:00:06.973017 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:06.972979    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f0206-695f4d5dc-594bn"]
Nov 01 23:00:08.172835 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.172800    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579fa21b-e969-40a4-bff2-29b2f9579a12" path="/var/lib/kubelet/pods/579fa21b-e969-40a4-bff2-29b2f9579a12/volumes"
Nov 01 23:00:08.642729 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.642684    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 23:00:08.642975 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.642949    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy" containerID="cri-o://50e5b04830385eff9e0b92cd2cc1a37cf329a402770c2f44006f2d4a73343b2d" gracePeriod=30
Nov 01 23:00:08.774296 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.774256    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 23:00:08.774602 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.774530    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy" containerID="cri-o://c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b" gracePeriod=30
Nov 01 23:00:08.821393 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.820270    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 23:00:08.821393 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:08.820622    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy" containerID="cri-o://cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26" gracePeriod=30
Nov 01 23:00:10.351260 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.351218    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"]
Nov 01 23:00:10.351948 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.351932    5031 cpu_manager.go:414] "containerMap: removing container" podUID="579fa21b-e969-40a4-bff2-29b2f9579a12" containerName="maas-api"
Nov 01 23:00:10.351998 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.351950    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579fa21b-e969-40a4-bff2-29b2f9579a12" containerName="maas-api"
Nov 01 23:00:10.352065 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.352055    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="579fa21b-e969-40a4-bff2-29b2f9579a12" containerName="maas-api"
Nov 01 23:00:10.357530 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.357510    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.507647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.507602    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"]
Nov 01 23:00:10.523876 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.523841    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:00:10.524054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.523899    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:00:10.540017 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.539983    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.540189 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.540108    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.540189 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.540133    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw84t\" (UniqueName: \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.628310 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:10.628213    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:00:10.642190 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.642157    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.642353 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.642260    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.642353 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.642285    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw84t\" (UniqueName: \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.642942 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.642922    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.644862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.644838    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.711083 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.711049    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw84t\" (UniqueName: \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\") pod \"maas-api-revoke-keys-e2e-api-a-71505e-q6mz6\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:10.973507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:10.973417    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:11.435752 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:11.435717    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"]
Nov 01 23:00:11.796007 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:11.795970    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6" event={"ID":"20e3f993-0f11-4f5a-86bd-b605ff10e6ef","Type":"ContainerStarted","Data":"915798eb52afb9c122237fe9ad947f6f34e8857d8425933b12809bfdfa6ef754"}
Nov 01 23:00:12.351051 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.351013    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 23:00:12.351335 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.351289    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" podUID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" containerName="maas-api" containerID="cri-o://2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe" gracePeriod=30
Nov 01 23:00:12.674656 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.674624    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 23:00:12.789940 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.789900    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\") pod \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") "
Nov 01 23:00:12.790121 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.790009    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\" (UniqueName: \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\") pod \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") "
Nov 01 23:00:12.790200 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.790184    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\") pod \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") "
Nov 01 23:00:12.790360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.790336    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") pod \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\" (UID: \"c094b927-ca3e-4e73-bd42-2b38968d7ce4\") "
Nov 01 23:00:12.790609 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.790577    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca" pod "c094b927-ca3e-4e73-bd42-2b38968d7ce4" (UID: "c094b927-ca3e-4e73-bd42-2b38968d7ce4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:12.791437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.791396    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c094b927-ca3e-4e73-bd42-2b38968d7ce4-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:12.793142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.793105    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs" pod "c094b927-ca3e-4e73-bd42-2b38968d7ce4" (UID: "c094b927-ca3e-4e73-bd42-2b38968d7ce4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:12.795512 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.795475    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466" pod "c094b927-ca3e-4e73-bd42-2b38968d7ce4" (UID: "c094b927-ca3e-4e73-bd42-2b38968d7ce4"). InnerVolumeSpecName "kube-api-access-g6466". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:12.797251 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.797216    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls" pod "c094b927-ca3e-4e73-bd42-2b38968d7ce4" (UID: "c094b927-ca3e-4e73-bd42-2b38968d7ce4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:12.808125 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.807885    5031 generic.go:356] "Generic (PLEG): container finished" podID="20e3f993-0f11-4f5a-86bd-b605ff10e6ef" containerID="f6147dc216ab220a5fe8abe251f82e6efa7cc3944cc58b1b132870b83e9c276e" exitCode=0
Nov 01 23:00:12.808125 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.808038    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6" event={"ID":"20e3f993-0f11-4f5a-86bd-b605ff10e6ef","Type":"ContainerDied","Data":"f6147dc216ab220a5fe8abe251f82e6efa7cc3944cc58b1b132870b83e9c276e"}
Nov 01 23:00:12.810554 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.810511    5031 generic.go:356] "Generic (PLEG): container finished" podID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" containerID="2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe" exitCode=0
Nov 01 23:00:12.811221 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.810786    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" event={"ID":"c094b927-ca3e-4e73-bd42-2b38968d7ce4","Type":"ContainerDied","Data":"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"}
Nov 01 23:00:12.811221 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.810875    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp" event={"ID":"c094b927-ca3e-4e73-bd42-2b38968d7ce4","Type":"ContainerDied","Data":"259fa04326a63425fc4cf4cb767b270795772035aa4bf4cc4cc7665d576312e1"}
Nov 01 23:00:12.811221 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.810943    5031 scope.go:122] "RemoveContainer" containerID="2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"
Nov 01 23:00:12.811987 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.811962    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"
Nov 01 23:00:12.841221 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.841199    5031 scope.go:122] "RemoveContainer" containerID="2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"
Nov 01 23:00:12.841705 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:12.841669    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe\": container with ID starting with 2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe not found: ID does not exist" containerID="2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"
Nov 01 23:00:12.841833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.841709    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe"} err="failed to get container status \"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe\": rpc error: code = NotFound desc = could not find container \"2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe\": container with ID starting with 2fc3a4f3a29ca56ddac00447614697f02aebd15a1f40d4d85dd3427c035469fe not found: ID does not exist"
Nov 01 23:00:12.892489 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.892407    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:12.892489 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.892435    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c094b927-ca3e-4e73-bd42-2b38968d7ce4-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:12.892489 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:12.892446    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6466\" (UniqueName: \"kubernetes.io/projected/c094b927-ca3e-4e73-bd42-2b38968d7ce4-kube-api-access-g6466\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:13.822939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:13.822900    5031 generic.go:356] "Generic (PLEG): container finished" podID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerID="50e5b04830385eff9e0b92cd2cc1a37cf329a402770c2f44006f2d4a73343b2d" exitCode=0
Nov 01 23:00:13.823372 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:13.822985    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" event={"ID":"b3886168-b1b1-42c6-9bd6-3d4e5b631253","Type":"ContainerDied","Data":"50e5b04830385eff9e0b92cd2cc1a37cf329a402770c2f44006f2d4a73343b2d"}
Nov 01 23:00:13.988951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:13.988921    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 23:00:14.009638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.009604    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.009807 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.009726    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.009807 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.009768    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.009971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.009951    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.010034 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.009991    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.010205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.010187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.010269 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.010239    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.010309 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.010281    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.010345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.010326    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\") pod \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\" (UID: \"b3886168-b1b1-42c6-9bd6-3d4e5b631253\") "
Nov 01 23:00:14.011390 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.011354    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.011800 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.011772    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.012062 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.012034    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.012273 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.012248    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.012492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.012470    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.015383 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.015344    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.021333 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.021205    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "kube-api-access-hcx9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.023231 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.023200    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.100823    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.106949    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy" pod "b3886168-b1b1-42c6-9bd6-3d4e5b631253" (UID: "b3886168-b1b1-42c6-9bd6-3d4e5b631253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114562    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114593    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcx9g\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-kube-api-access-hcx9g\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114622    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114634    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114646    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114659    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114671    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114695    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3886168-b1b1-42c6-9bd6-3d4e5b631253-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.121592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.114706    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3886168-b1b1-42c6-9bd6-3d4e5b631253-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.211887 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.211838    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:14.237147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.237112    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f5b935-56767cddfd-mpfrp"]
Nov 01 23:00:14.298647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.298601    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 23:00:14.316282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.316246    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 23:00:14.316753 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.316719    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.316884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.316790    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.316884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.316876    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\" (UniqueName: \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\") pod \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") "
Nov 01 23:00:14.316982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.316949    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317075 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317049    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.317158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317108    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317142    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.317259 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317191    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317259 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317249    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317365 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317323    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\") pod \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") "
Nov 01 23:00:14.317365 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317355    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\") pod \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\" (UID: \"20e3f993-0f11-4f5a-86bd-b605ff10e6ef\") "
Nov 01 23:00:14.317468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317386    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317442    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.317633 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.317570    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\") pod \"ca60c084-efcc-4e56-a268-1cc20fce1c40\" (UID: \"ca60c084-efcc-4e56-a268-1cc20fce1c40\") "
Nov 01 23:00:14.318139 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.318110    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.318423 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.318398    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.318721 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.318679    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca" pod "20e3f993-0f11-4f5a-86bd-b605ff10e6ef" (UID: "20e3f993-0f11-4f5a-86bd-b605ff10e6ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.318911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.318847    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.319449 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319420    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-openshift-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.319568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319455    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.319568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319471    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca60c084-efcc-4e56-a268-1cc20fce1c40-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.319568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319485    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.319568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319500    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.319568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.319515    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.320605 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.320563    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.320825 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.320806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t" pod "20e3f993-0f11-4f5a-86bd-b605ff10e6ef" (UID: "20e3f993-0f11-4f5a-86bd-b605ff10e6ef"). InnerVolumeSpecName "kube-api-access-hw84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.320825 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.320806    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca" pod "20e3f993-0f11-4f5a-86bd-b605ff10e6ef" (UID: "20e3f993-0f11-4f5a-86bd-b605ff10e6ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:14.321629 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.321609    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.322282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.322264    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "kube-api-access-6gxf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.324862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.324831    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo" pod "ca60c084-efcc-4e56-a268-1cc20fce1c40" (UID: "ca60c084-efcc-4e56-a268-1cc20fce1c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.420440 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420398    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.420686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420479    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.420686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420614    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420709    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420782    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420960    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.420999    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421038    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421072    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\") pod \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\" (UID: \"b503ae4b-0ac0-434f-a8e3-2b728724ef3a\") "
Nov 01 23:00:14.421783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421763    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw84t\" (UniqueName: \"kubernetes.io/projected/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-kube-api-access-hw84t\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.421885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421786    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.421885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421797    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.421885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421807    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca60c084-efcc-4e56-a268-1cc20fce1c40-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.421885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421817    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gxf7\" (UniqueName: \"kubernetes.io/projected/ca60c084-efcc-4e56-a268-1cc20fce1c40-kube-api-access-6gxf7\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.421885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.421827    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e3f993-0f11-4f5a-86bd-b605ff10e6ef-maas-api-xks-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.423299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.422988    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.426366 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.426323    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.427507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.426861    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.427507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.427168    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.427507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.427378    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.427507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.427467    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.429315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.429272    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "kube-api-access-fsvpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.430959 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.430923    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.435143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.435109    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token" pod "b503ae4b-0ac0-434f-a8e3-2b728724ef3a" (UID: "b503ae4b-0ac0-434f-a8e3-2b728724ef3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523732    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523772    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523782    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523791    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523802    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523811    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.523823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523821    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.524158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523837    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvpz\" (UniqueName: \"kubernetes.io/projected/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-kube-api-access-fsvpz\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.524158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.523848    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b503ae4b-0ac0-434f-a8e3-2b728724ef3a-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.831666 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.831619    5031 generic.go:356] "Generic (PLEG): container finished" podID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerID="c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b" exitCode=0
Nov 01 23:00:14.832162 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.831686    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"
Nov 01 23:00:14.832162 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.831730    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" event={"ID":"ca60c084-efcc-4e56-a268-1cc20fce1c40","Type":"ContainerDied","Data":"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"}
Nov 01 23:00:14.832162 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.831788    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp" event={"ID":"ca60c084-efcc-4e56-a268-1cc20fce1c40","Type":"ContainerDied","Data":"c7ba06b46171aa91491f471064b960f9bfa6326c7334a9a4a33e3be352ff3fde"}
Nov 01 23:00:14.832162 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.831803    5031 scope.go:122] "RemoveContainer" containerID="c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"
Nov 01 23:00:14.833513 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.833476    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"
Nov 01 23:00:14.833513 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.833494    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7" event={"ID":"b3886168-b1b1-42c6-9bd6-3d4e5b631253","Type":"ContainerDied","Data":"0a5ed702bb9564845657447838ac402e97cafae38d841f9d34b5b76aad09d577"}
Nov 01 23:00:14.835386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.835357    5031 generic.go:356] "Generic (PLEG): container finished" podID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerID="cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26" exitCode=0
Nov 01 23:00:14.836048 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.835501    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" event={"ID":"b503ae4b-0ac0-434f-a8e3-2b728724ef3a","Type":"ContainerDied","Data":"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"}
Nov 01 23:00:14.836048 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.835601    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq" event={"ID":"b503ae4b-0ac0-434f-a8e3-2b728724ef3a","Type":"ContainerDied","Data":"15cc14fb8258ca9284aed03ef07b93605438de0f57c2c985eb0dd0525bbb257f"}
Nov 01 23:00:14.836048 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.835702    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"
Nov 01 23:00:14.841493 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.841465    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"
Nov 01 23:00:14.841493 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.841479    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6" event={"ID":"20e3f993-0f11-4f5a-86bd-b605ff10e6ef","Type":"ContainerDied","Data":"915798eb52afb9c122237fe9ad947f6f34e8857d8425933b12809bfdfa6ef754"}
Nov 01 23:00:14.841734 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.841524    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="915798eb52afb9c122237fe9ad947f6f34e8857d8425933b12809bfdfa6ef754"
Nov 01 23:00:14.846275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.846248    5031 scope.go:122] "RemoveContainer" containerID="c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"
Nov 01 23:00:14.847424 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:14.846914    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b\": container with ID starting with c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b not found: ID does not exist" containerID="c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"
Nov 01 23:00:14.847424 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.846957    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b"} err="failed to get container status \"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b\": rpc error: code = NotFound desc = could not find container \"c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b\": container with ID starting with c0809fef15ca91d20483d332327adabfa188708d832e8bba312cf80194dcfe4b not found: ID does not exist"
Nov 01 23:00:14.847424 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.846983    5031 scope.go:122] "RemoveContainer" containerID="50e5b04830385eff9e0b92cd2cc1a37cf329a402770c2f44006f2d4a73343b2d"
Nov 01 23:00:14.860744 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.860718    5031 scope.go:122] "RemoveContainer" containerID="cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"
Nov 01 23:00:14.872681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.872653    5031 scope.go:122] "RemoveContainer" containerID="cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"
Nov 01 23:00:14.873055 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:14.873027    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26\": container with ID starting with cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26 not found: ID does not exist" containerID="cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"
Nov 01 23:00:14.873147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:14.873065    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26"} err="failed to get container status \"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26\": rpc error: code = NotFound desc = could not find container \"cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26\": container with ID starting with cf849145a761078c93da2e6f01f51ed9db6714717537be91e97b7e7b919d9c26 not found: ID does not exist"
Nov 01 23:00:16.172866 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:16.172784    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" path="/var/lib/kubelet/pods/c094b927-ca3e-4e73-bd42-2b38968d7ce4/volumes"
Nov 01 23:00:17.600856 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:17.600821    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 23:00:17.667090 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:17.667048    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-f7mxp"]
Nov 01 23:00:17.838468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:17.838431    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 23:00:17.953467 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:17.953374    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-g22dq"]
Nov 01 23:00:18.172355 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:18.172323    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" path="/var/lib/kubelet/pods/b503ae4b-0ac0-434f-a8e3-2b728724ef3a/volumes"
Nov 01 23:00:18.172864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:18.172849    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" path="/var/lib/kubelet/pods/ca60c084-efcc-4e56-a268-1cc20fce1c40/volumes"
Nov 01 23:00:18.449033 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:18.449000    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 23:00:18.622636 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:18.622593    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-w7jn7"]
Nov 01 23:00:20.180903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:20.180858    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" path="/var/lib/kubelet/pods/b3886168-b1b1-42c6-9bd6-3d4e5b631253/volumes"
Nov 01 23:00:22.803459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:22.803420    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 23:00:24.233626 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:24.233585    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 23:00:24.234096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:24.233860    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy" containerID="cri-o://e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a" gracePeriod=30
Nov 01 23:00:25.641282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.641246    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 23:00:25.641788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.641455    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" podUID="5f9910a1-b517-4572-9a16-17da3b92450b" containerName="maas-api" containerID="cri-o://9f82af32a6385931d7f1e7c4e955e1fe94464ec7e2f999be6e312f623d7040b1" gracePeriod=30
Nov 01 23:00:25.940678 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.940644    5031 generic.go:356] "Generic (PLEG): container finished" podID="5f9910a1-b517-4572-9a16-17da3b92450b" containerID="9f82af32a6385931d7f1e7c4e955e1fe94464ec7e2f999be6e312f623d7040b1" exitCode=0
Nov 01 23:00:25.940848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.940715    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" event={"ID":"5f9910a1-b517-4572-9a16-17da3b92450b","Type":"ContainerDied","Data":"9f82af32a6385931d7f1e7c4e955e1fe94464ec7e2f999be6e312f623d7040b1"}
Nov 01 23:00:25.940848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.940769    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n" event={"ID":"5f9910a1-b517-4572-9a16-17da3b92450b","Type":"ContainerDied","Data":"4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c"}
Nov 01 23:00:25.940848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.940789    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a578d94bba87a19daf643fa3850810fd229171fb85bf8601dfd0cb39ee2af7c"
Nov 01 23:00:25.957318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:25.957290    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 23:00:26.133303 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.133262    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") pod \"5f9910a1-b517-4572-9a16-17da3b92450b\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") "
Nov 01 23:00:26.133508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.133312    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\") pod \"5f9910a1-b517-4572-9a16-17da3b92450b\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") "
Nov 01 23:00:26.133508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.133341    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\") pod \"5f9910a1-b517-4572-9a16-17da3b92450b\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") "
Nov 01 23:00:26.133508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.133385    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\" (UniqueName: \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\") pod \"5f9910a1-b517-4572-9a16-17da3b92450b\" (UID: \"5f9910a1-b517-4572-9a16-17da3b92450b\") "
Nov 01 23:00:26.134095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.134061    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca" pod "5f9910a1-b517-4572-9a16-17da3b92450b" (UID: "5f9910a1-b517-4572-9a16-17da3b92450b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:26.136922 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.136885    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2" pod "5f9910a1-b517-4572-9a16-17da3b92450b" (UID: "5f9910a1-b517-4572-9a16-17da3b92450b"). InnerVolumeSpecName "kube-api-access-x6mq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:26.138171 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.138122    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls" pod "5f9910a1-b517-4572-9a16-17da3b92450b" (UID: "5f9910a1-b517-4572-9a16-17da3b92450b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:26.139934 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.139891    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs" pod "5f9910a1-b517-4572-9a16-17da3b92450b" (UID: "5f9910a1-b517-4572-9a16-17da3b92450b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:26.235521 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.235482    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:26.235521 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.235521    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9910a1-b517-4572-9a16-17da3b92450b-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:26.235846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.235604    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9910a1-b517-4572-9a16-17da3b92450b-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:26.235846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.235618    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mq2\" (UniqueName: \"kubernetes.io/projected/5f9910a1-b517-4572-9a16-17da3b92450b-kube-api-access-x6mq2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:26.970729 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:26.968043    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"
Nov 01 23:00:27.152630 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:27.152587    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 23:00:27.197681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:27.197637    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-71505e-69c8f8bbbb-9vv5n"]
Nov 01 23:00:27.977848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:27.977816    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"]
Nov 01 23:00:28.023648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:28.023605    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71505e-q6mz6"]
Nov 01 23:00:28.173082 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:28.173044    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e3f993-0f11-4f5a-86bd-b605ff10e6ef" path="/var/lib/kubelet/pods/20e3f993-0f11-4f5a-86bd-b605ff10e6ef/volumes"
Nov 01 23:00:28.173483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:28.173468    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9910a1-b517-4572-9a16-17da3b92450b" path="/var/lib/kubelet/pods/5f9910a1-b517-4572-9a16-17da3b92450b/volumes"
Nov 01 23:00:29.520995 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.520964    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 23:00:29.694691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.694655    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.694846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.694712    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.694911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.694834    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.694968 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.694944    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695062    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695092    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695149    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695175    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695192    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695101    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695210    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:29.695244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695228    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695255    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\") pod \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\" (UID: \"3cf161fd-3ba6-4dab-baa4-be6e29b75c35\") "
Nov 01 23:00:29.695933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695695    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:29.695933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695910    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.695933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695932    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.696152 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695945    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.696152 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695957    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.696152 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.695968    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.697648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.697622    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:29.697839 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.697814    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:29.698285 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.698261    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "kube-api-access-4ksgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:29.698378 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.698297    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token" pod "3cf161fd-3ba6-4dab-baa4-be6e29b75c35" (UID: "3cf161fd-3ba6-4dab-baa4-be6e29b75c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:29.797167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.797122    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.797167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.797164    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.797381 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.797177    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ksgf\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-kube-api-access-4ksgf\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.797381 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.797191    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf161fd-3ba6-4dab-baa4-be6e29b75c35-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:29.991423 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.991392    5031 generic.go:356] "Generic (PLEG): container finished" podID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerID="e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a" exitCode=0
Nov 01 23:00:29.991648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.991445    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"
Nov 01 23:00:29.991648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.991485    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerDied","Data":"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"}
Nov 01 23:00:29.991648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.991575    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t" event={"ID":"3cf161fd-3ba6-4dab-baa4-be6e29b75c35","Type":"ContainerDied","Data":"29afea8ab7d59fdb4a5b04ecc28d53c58b984da315b22147c41c5202e8de9e53"}
Nov 01 23:00:29.991648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:29.991600    5031 scope.go:122] "RemoveContainer" containerID="e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"
Nov 01 23:00:30.006413 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.006386    5031 scope.go:122] "RemoveContainer" containerID="96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"
Nov 01 23:00:30.016142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.016106    5031 scope.go:122] "RemoveContainer" containerID="e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"
Nov 01 23:00:30.016428 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:30.016408    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a\": container with ID starting with e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a not found: ID does not exist" containerID="e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"
Nov 01 23:00:30.016496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.016443    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a"} err="failed to get container status \"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a\": rpc error: code = NotFound desc = could not find container \"e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a\": container with ID starting with e476d5783c389c2b6242cff449888382bac9c6fd63d0d84bd8e99b0d6f883a7a not found: ID does not exist"
Nov 01 23:00:30.016496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.016471    5031 scope.go:122] "RemoveContainer" containerID="96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"
Nov 01 23:00:30.016753 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:30.016736    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6\": container with ID starting with 96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6 not found: ID does not exist" containerID="96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"
Nov 01 23:00:30.016810 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.016760    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6"} err="failed to get container status \"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6\": rpc error: code = NotFound desc = could not find container \"96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6\": container with ID starting with 96de3d22c06b9f85a7bb1a3536719c1591de8ab92b416fecad03aee6649362a6 not found: ID does not exist"
Nov 01 23:00:30.192502 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.192406    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 23:00:30.244640 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:30.244592    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d68bab-openshift-default-65d6957b67-mlv2t"]
Nov 01 23:00:32.224999 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:32.221169    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" path="/var/lib/kubelet/pods/3cf161fd-3ba6-4dab-baa4-be6e29b75c35/volumes"
Nov 01 23:00:32.680238 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:32.680184    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 23:00:32.681393 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:32.681147    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" containerID="cri-o://79cf2e6d9085f8ebfb3abcede140ad03e67c87145a9b11f79c23230f64ebd6cd" gracePeriod=30
Nov 01 23:00:32.729861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:32.729809    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 23:00:32.730409 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:32.730327    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" containerID="cri-o://068d9ad0a289cfcf8aa9edd09ef9e9b70da31e8a5b3edb7a3f5718612075882a" gracePeriod=30
Nov 01 23:00:33.890638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:33.890599    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 23:00:33.891068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:33.890854    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" containerID="cri-o://e346eaca0820214d049bd081c37c6828a9c569b0a0001e69a7f8934f57e25238" gracePeriod=30
Nov 01 23:00:34.052260 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:34.052223    5031 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:00:34.052416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:34.052295    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:00:35.869491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:35.869455    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 23:00:37.414003 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:37.413262    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:00:37.414003 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:37.413350    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:00:37.826461 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:37.826421    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" containerID="cri-o://a618ddc0605cc07e988003575d4e133512d6d6a49d8f1c150ab464f9fade084b" gracePeriod=45
Nov 01 23:00:38.079954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.079846    5031 generic.go:356] "Generic (PLEG): container finished" podID="c8202dac-7445-45b8-91f3-afdac802965d" containerID="79cf2e6d9085f8ebfb3abcede140ad03e67c87145a9b11f79c23230f64ebd6cd" exitCode=0
Nov 01 23:00:38.080225 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.080047    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" event={"ID":"c8202dac-7445-45b8-91f3-afdac802965d","Type":"ContainerDied","Data":"79cf2e6d9085f8ebfb3abcede140ad03e67c87145a9b11f79c23230f64ebd6cd"}
Nov 01 23:00:38.086884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.086815    5031 generic.go:356] "Generic (PLEG): container finished" podID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerID="a618ddc0605cc07e988003575d4e133512d6d6a49d8f1c150ab464f9fade084b" exitCode=0
Nov 01 23:00:38.087067 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.086955    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" event={"ID":"d48b3c0a-9e20-40f5-b441-4109ded549c9","Type":"ContainerDied","Data":"a618ddc0605cc07e988003575d4e133512d6d6a49d8f1c150ab464f9fade084b"}
Nov 01 23:00:38.089607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.089573    5031 generic.go:356] "Generic (PLEG): container finished" podID="755e265e-0fc2-40d5-a91a-62c293471873" containerID="068d9ad0a289cfcf8aa9edd09ef9e9b70da31e8a5b3edb7a3f5718612075882a" exitCode=0
Nov 01 23:00:38.089769 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.089694    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" event={"ID":"755e265e-0fc2-40d5-a91a-62c293471873","Type":"ContainerDied","Data":"068d9ad0a289cfcf8aa9edd09ef9e9b70da31e8a5b3edb7a3f5718612075882a"}
Nov 01 23:00:38.127214 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.127186    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 23:00:38.140932 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.140904    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 23:00:38.202698 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.202663    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.202698 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.202705    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.202920 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.202734    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.202920 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.202795    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\" (UniqueName: \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.202920 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.202810    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.203056 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203007    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.203056 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203034    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.203172 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203071    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203232 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203169    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203283 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203231    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203283 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203276    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203396 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203310    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203396 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203315    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.203396 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203380    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.203567 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203387    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203567 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203510    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.203701 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203528    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.203701 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203492    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203816 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203777    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\") pod \"755e265e-0fc2-40d5-a91a-62c293471873\" (UID: \"755e265e-0fc2-40d5-a91a-62c293471873\") "
Nov 01 23:00:38.203903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203879    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\") pod \"d48b3c0a-9e20-40f5-b441-4109ded549c9\" (UID: \"d48b3c0a-9e20-40f5-b441-4109ded549c9\") "
Nov 01 23:00:38.203964 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.203899    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:38.204585 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204522    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204626    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755e265e-0fc2-40d5-a91a-62c293471873-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204650    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204663    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204677    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204690    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204628    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.204880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.204703    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.205937 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.205908    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.206063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.205940    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.206063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.205996    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:38.206185 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.206171    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "kube-api-access-j6pbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.206457 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.206441    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.206690 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.206665    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj" pod "d48b3c0a-9e20-40f5-b441-4109ded549c9" (UID: "d48b3c0a-9e20-40f5-b441-4109ded549c9"). InnerVolumeSpecName "kube-api-access-84vsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.206818 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.206804    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo" pod "755e265e-0fc2-40d5-a91a-62c293471873" (UID: "755e265e-0fc2-40d5-a91a-62c293471873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306110    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306142    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755e265e-0fc2-40d5-a91a-62c293471873-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306155    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pbn\" (UniqueName: \"kubernetes.io/projected/755e265e-0fc2-40d5-a91a-62c293471873-kube-api-access-j6pbn\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306165    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306174    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306183    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48b3c0a-9e20-40f5-b441-4109ded549c9-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306192    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755e265e-0fc2-40d5-a91a-62c293471873-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306206    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84vsj\" (UniqueName: \"kubernetes.io/projected/d48b3c0a-9e20-40f5-b441-4109ded549c9-kube-api-access-84vsj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.306247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.306219    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48b3c0a-9e20-40f5-b441-4109ded549c9-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.370358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.370324    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 23:00:38.508202 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508161    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508241    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508320    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508375    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508434    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508498    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508524    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508610    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508649    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.508711 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508686    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\") pod \"c8202dac-7445-45b8-91f3-afdac802965d\" (UID: \"c8202dac-7445-45b8-91f3-afdac802965d\") "
Nov 01 23:00:38.509195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.509195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.508957    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:38.509930 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.509866    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.510992 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.510939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.511883 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511858    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511887    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511901    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511913    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511926    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8202dac-7445-45b8-91f3-afdac802965d-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.511940    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.512118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.512091    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "kube-api-access-z98tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:38.512926 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.512906    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:38.513383 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.513359    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo" pod "c8202dac-7445-45b8-91f3-afdac802965d" (UID: "c8202dac-7445-45b8-91f3-afdac802965d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:38.613442 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.613405    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.613442 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.613438    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8202dac-7445-45b8-91f3-afdac802965d-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.613442 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.613448    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z98tx\" (UniqueName: \"kubernetes.io/projected/c8202dac-7445-45b8-91f3-afdac802965d-kube-api-access-z98tx\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:38.613708 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:38.613460    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8202dac-7445-45b8-91f3-afdac802965d-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.011612 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.011571    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:00:39.011755 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.011654    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:39.074695 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.074646    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:39.107448 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.107395    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk" event={"ID":"755e265e-0fc2-40d5-a91a-62c293471873","Type":"ContainerDied","Data":"283455c983352435be50f87e4cd1a6d57da863db6ca055b2ac9e05252c4e515c"}
Nov 01 23:00:39.107685 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.107501    5031 scope.go:122] "RemoveContainer" containerID="068d9ad0a289cfcf8aa9edd09ef9e9b70da31e8a5b3edb7a3f5718612075882a"
Nov 01 23:00:39.107854 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.107827    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"
Nov 01 23:00:39.110057 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.110030    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd" event={"ID":"c8202dac-7445-45b8-91f3-afdac802965d","Type":"ContainerDied","Data":"b7b44644eb4b41c7aac364c5a897ba92d84b2653f97d1765463afc3220224cf7"}
Nov 01 23:00:39.110177 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.110142    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"
Nov 01 23:00:39.113679 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.113643    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb" event={"ID":"d48b3c0a-9e20-40f5-b441-4109ded549c9","Type":"ContainerDied","Data":"61479aed73057a2a2bb518d0f5477f92e66d0e30df60e839969a45313a8e3c4c"}
Nov 01 23:00:39.113840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.113755    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"
Nov 01 23:00:39.118990 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.118946    5031 generic.go:356] "Generic (PLEG): container finished" podID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerID="e346eaca0820214d049bd081c37c6828a9c569b0a0001e69a7f8934f57e25238" exitCode=0
Nov 01 23:00:39.118990 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.118989    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" event={"ID":"766d5209-2eac-4d97-81bc-4a5b9f9d8608","Type":"ContainerDied","Data":"e346eaca0820214d049bd081c37c6828a9c569b0a0001e69a7f8934f57e25238"}
Nov 01 23:00:39.131845 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.131816    5031 scope.go:122] "RemoveContainer" containerID="79cf2e6d9085f8ebfb3abcede140ad03e67c87145a9b11f79c23230f64ebd6cd"
Nov 01 23:00:39.145719 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.144036    5031 scope.go:122] "RemoveContainer" containerID="a618ddc0605cc07e988003575d4e133512d6d6a49d8f1c150ab464f9fade084b"
Nov 01 23:00:39.211253 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.211193    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 23:00:39.288762 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.288727    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 23:00:39.322581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322527    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.322783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322635    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.322783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322687    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.322783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.322962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322847    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.322962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322935    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.323073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.322970    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.323073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323005    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.323073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323044    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.323211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323084    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\") pod \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\" (UID: \"766d5209-2eac-4d97-81bc-4a5b9f9d8608\") "
Nov 01 23:00:39.323211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323086    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.323307 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323261    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.323781 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323299    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.323781 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323476    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:39.323979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323780    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.323979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323801    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.323979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323816    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.323979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323831    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.323979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.323845    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.325144 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.325124    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:39.325449 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.325433    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:39.325789 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.325770    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:39.325873 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.325809    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws" pod "766d5209-2eac-4d97-81bc-4a5b9f9d8608" (UID: "766d5209-2eac-4d97-81bc-4a5b9f9d8608"). InnerVolumeSpecName "kube-api-access-m9dws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:39.335709 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.335572    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-glzxd"]
Nov 01 23:00:39.425422 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.425338    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.425422 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.425370    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.425422 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.425381    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766d5209-2eac-4d97-81bc-4a5b9f9d8608-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.425422 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.425393    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dws\" (UniqueName: \"kubernetes.io/projected/766d5209-2eac-4d97-81bc-4a5b9f9d8608-kube-api-access-m9dws\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:39.449024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.448983    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 23:00:39.453321 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.453287    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b8fd2/distinct-w1-5b8fd2-kserve-7bb5fc4cd7-zgbcb"]
Nov 01 23:00:39.555279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.555241    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 23:00:39.601335 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:39.601300    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-v5kfk"]
Nov 01 23:00:40.128386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.128353    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"
Nov 01 23:00:40.128386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.128361    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds" event={"ID":"766d5209-2eac-4d97-81bc-4a5b9f9d8608","Type":"ContainerDied","Data":"b435ef3a1dd449548516e2e79f8446fbea47307fa0f8f3c149467607f3142a50"}
Nov 01 23:00:40.128676 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.128459    5031 scope.go:122] "RemoveContainer" containerID="e346eaca0820214d049bd081c37c6828a9c569b0a0001e69a7f8934f57e25238"
Nov 01 23:00:40.173839 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.173799    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755e265e-0fc2-40d5-a91a-62c293471873" path="/var/lib/kubelet/pods/755e265e-0fc2-40d5-a91a-62c293471873/volumes"
Nov 01 23:00:40.175223 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.174613    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8202dac-7445-45b8-91f3-afdac802965d" path="/var/lib/kubelet/pods/c8202dac-7445-45b8-91f3-afdac802965d/volumes"
Nov 01 23:00:40.175385 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.175265    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" path="/var/lib/kubelet/pods/d48b3c0a-9e20-40f5-b441-4109ded549c9/volumes"
Nov 01 23:00:40.294606 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.294562    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:00:40.295311 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295288    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy"
Nov 01 23:00:40.295311 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295310    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy"
Nov 01 23:00:40.295311 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295319    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" containerName="maas-api"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295324    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" containerName="maas-api"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295335    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295340    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295351    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5f9910a1-b517-4572-9a16-17da3b92450b" containerName="maas-api"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295355    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9910a1-b517-4572-9a16-17da3b92450b" containerName="maas-api"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295375    5031 cpu_manager.go:414] "containerMap: removing container" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295380    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295390    5031 cpu_manager.go:414] "containerMap: removing container" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295394    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295401    5031 cpu_manager.go:414] "containerMap: removing container" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295407    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295425    5031 cpu_manager.go:414] "containerMap: removing container" podUID="20e3f993-0f11-4f5a-86bd-b605ff10e6ef" containerName="revoke-keys"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295430    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e3f993-0f11-4f5a-86bd-b605ff10e6ef" containerName="revoke-keys"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295439    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295443    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295453    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295457    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295476    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295480    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy"
Nov 01 23:00:40.295604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295616    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca60c084-efcc-4e56-a268-1cc20fce1c40" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295629    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b503ae4b-0ac0-434f-a8e3-2b728724ef3a" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295644    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c094b927-ca3e-4e73-bd42-2b38968d7ce4" containerName="maas-api"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295651    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295658    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9910a1-b517-4572-9a16-17da3b92450b" containerName="maas-api"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295668    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295674    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="755e265e-0fc2-40d5-a91a-62c293471873" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295681    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8202dac-7445-45b8-91f3-afdac802965d" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295687    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295693    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e3f993-0f11-4f5a-86bd-b605ff10e6ef" containerName="revoke-keys"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295704    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3886168-b1b1-42c6-9bd6-3d4e5b631253" containerName="istio-proxy"
Nov 01 23:00:40.296707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.295711    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48b3c0a-9e20-40f5-b441-4109ded549c9" containerName="main"
Nov 01 23:00:40.301389 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.301361    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.302872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.302847    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 23:00:40.340466 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.340427    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-2l4ds"]
Nov 01 23:00:40.360799 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.360758    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-675d1d\""
Nov 01 23:00:40.364859 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.364816    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:00:40.383495 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.383404    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-675d1d-dockercfg-z25bq\""
Nov 01 23:00:40.470638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.465515    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.470638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.465732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl452\" (UniqueName: \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.475488 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.475460    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:00:40.476122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.476099    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.476122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.476117    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf161fd-3ba6-4dab-baa4-be6e29b75c35" containerName="istio-proxy"
Nov 01 23:00:40.479706 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.479687    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.496576 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.496525    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:00:40.567781 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.567744    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl452\" (UniqueName: \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.568257 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.567925    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.568257 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.567983    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjj7\" (UniqueName: \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.568257 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.568046    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.568907 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.568882    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.622316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.622278    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl452\" (UniqueName: \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\") pod \"payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:40.630987 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:00:40.630945    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:00:40.668935 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.668835    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjj7\" (UniqueName: \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.668935 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.668899    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.669626 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.669604    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.723267 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.723234    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjj7\" (UniqueName: \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\") pod \"payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.800395 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.800354    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:40.869962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.869902    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" podUID="29557792-57fd-4908-8b2a-04dc5672b492" containerName="payload-pre-processing" containerID="cri-o://fadea0a8c34f7fd705d39980dea9521aa69ffb4ed00c24a4c11263d798c32b2a" gracePeriod=25
Nov 01 23:00:40.916994 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:40.916959    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:41.148836 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.148608    5031 generic.go:356] "Generic (PLEG): container finished" podID="29557792-57fd-4908-8b2a-04dc5672b492" containerID="fadea0a8c34f7fd705d39980dea9521aa69ffb4ed00c24a4c11263d798c32b2a" exitCode=0
Nov 01 23:00:41.148836 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.148709    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" event={"ID":"29557792-57fd-4908-8b2a-04dc5672b492","Type":"ContainerDied","Data":"fadea0a8c34f7fd705d39980dea9521aa69ffb4ed00c24a4c11263d798c32b2a"}
Nov 01 23:00:41.192308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.190620    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:00:41.196401 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:00:41.196360    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5038f7_f7e4_41e9_94c0_d7fc54cc80ea.slice/crio-b0a83e9455d2b5ab6d1f6bc6a4ff6c22e8725e6dd39dfeb59f569bbd16a9e42f WatchSource:0}: Error finding container b0a83e9455d2b5ab6d1f6bc6a4ff6c22e8725e6dd39dfeb59f569bbd16a9e42f: Status 404 returned error can't find the container with id b0a83e9455d2b5ab6d1f6bc6a4ff6c22e8725e6dd39dfeb59f569bbd16a9e42f
Nov 01 23:00:41.221767 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.221743    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 23:00:41.276594 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.276567    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\" (UniqueName: \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\") pod \"29557792-57fd-4908-8b2a-04dc5672b492\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") "
Nov 01 23:00:41.276714 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.276624    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\") pod \"29557792-57fd-4908-8b2a-04dc5672b492\" (UID: \"29557792-57fd-4908-8b2a-04dc5672b492\") "
Nov 01 23:00:41.277087 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.277059    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume" pod "29557792-57fd-4908-8b2a-04dc5672b492" (UID: "29557792-57fd-4908-8b2a-04dc5672b492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:41.277338 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.277313    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29557792-57fd-4908-8b2a-04dc5672b492-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.278845 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.278818    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t" pod "29557792-57fd-4908-8b2a-04dc5672b492" (UID: "29557792-57fd-4908-8b2a-04dc5672b492"). InnerVolumeSpecName "kube-api-access-vgk8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:41.311232 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.311192    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:00:41.316239 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:00:41.316209    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c305d46_d28a_4b6b_90eb_9663bd155d17.slice/crio-622c6f40f1fd97b2a74628737d5f47cfb5526b12f8bc2ec4de512e9b32906513 WatchSource:0}: Error finding container 622c6f40f1fd97b2a74628737d5f47cfb5526b12f8bc2ec4de512e9b32906513: Status 404 returned error can't find the container with id 622c6f40f1fd97b2a74628737d5f47cfb5526b12f8bc2ec4de512e9b32906513
Nov 01 23:00:41.378444 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:41.378410    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgk8t\" (UniqueName: \"kubernetes.io/projected/29557792-57fd-4908-8b2a-04dc5672b492-kube-api-access-vgk8t\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:42.158605 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.158574    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"
Nov 01 23:00:42.159105 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.158602    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg" event={"ID":"29557792-57fd-4908-8b2a-04dc5672b492","Type":"ContainerDied","Data":"a08b506a895ec322636d6d9b1ed3061590b8e9e5f2f3ca334659dabdc67b7089"}
Nov 01 23:00:42.159105 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.158670    5031 scope.go:122] "RemoveContainer" containerID="fadea0a8c34f7fd705d39980dea9521aa69ffb4ed00c24a4c11263d798c32b2a"
Nov 01 23:00:42.160903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.160857    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" event={"ID":"2c305d46-d28a-4b6b-90eb-9663bd155d17","Type":"ContainerStarted","Data":"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"}
Nov 01 23:00:42.161029 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.160939    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:00:42.161029 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.160966    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" event={"ID":"2c305d46-d28a-4b6b-90eb-9663bd155d17","Type":"ContainerStarted","Data":"622c6f40f1fd97b2a74628737d5f47cfb5526b12f8bc2ec4de512e9b32906513"}
Nov 01 23:00:42.162739 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.162710    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" event={"ID":"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea","Type":"ContainerStarted","Data":"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"}
Nov 01 23:00:42.162852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.162764    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" event={"ID":"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea","Type":"ContainerStarted","Data":"b0a83e9455d2b5ab6d1f6bc6a4ff6c22e8725e6dd39dfeb59f569bbd16a9e42f"}
Nov 01 23:00:42.162903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.162891    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:42.173071 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.173041    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766d5209-2eac-4d97-81bc-4a5b9f9d8608" path="/var/lib/kubelet/pods/766d5209-2eac-4d97-81bc-4a5b9f9d8608/volumes"
Nov 01 23:00:42.318998 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.318962    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 23:00:42.364522 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.364484    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-103369-c7c4f4f5f-br5xg"]
Nov 01 23:00:42.429167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.429048    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" podStartSLOduration=2.429032126 podStartE2EDuration="2.429032126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:42.426716126 +0000 UTC m=+2196.382755969" watchObservedRunningTime="2026-11-01 23:00:42.429032126 +0000 UTC m=+2196.385071969"
Nov 01 23:00:42.536066 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:42.535996    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" podStartSLOduration=2.535975827 podStartE2EDuration="2.535975827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:42.533146945 +0000 UTC m=+2196.489186835" watchObservedRunningTime="2026-11-01 23:00:42.535975827 +0000 UTC m=+2196.492015678"
Nov 01 23:00:44.174001 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:44.173965    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29557792-57fd-4908-8b2a-04dc5672b492" path="/var/lib/kubelet/pods/29557792-57fd-4908-8b2a-04dc5672b492/volumes"
Nov 01 23:00:53.176801 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:53.176771    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:00:53.177322 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:00:53.176851    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:01:10.628092 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:01:10.628051    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:01:18.478666 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:18.478631    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 23:01:18.479109 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:18.478859    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy" containerID="cri-o://e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd" gracePeriod=30
Nov 01 23:01:23.773672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.773638    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 23:01:23.903062 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903029    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903073    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903186    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903230    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903266    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903312    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903398    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903438    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903462    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:01:23.903496 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903490    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\") pod \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\" (UID: \"1d84ff32-5ab1-4588-b7de-a97ae8f70385\") "
Nov 01 23:01:23.903825 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903695    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:01:23.903825 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.903809    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:23.904154 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904011    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:01:23.904154 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904042    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:01:23.904517 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904495    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.904642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904522    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.904642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904556    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.904642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904571    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.904642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.904586    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.905675 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.905648    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:01:23.905675 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.905661    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "kube-api-access-9lp5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:23.905821 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.905715    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:23.905862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:23.905835    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy" pod "1d84ff32-5ab1-4588-b7de-a97ae8f70385" (UID: "1d84ff32-5ab1-4588-b7de-a97ae8f70385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:01:24.005805 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.005722    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lp5l\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-kube-api-access-9lp5l\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:24.005805 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.005751    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:24.005805 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.005760    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:24.005805 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.005769    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d84ff32-5ab1-4588-b7de-a97ae8f70385-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:24.458851 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.458816    5031 generic.go:356] "Generic (PLEG): container finished" podID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerID="e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd" exitCode=0
Nov 01 23:01:24.459033 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.458871    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"
Nov 01 23:01:24.459033 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.458894    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerDied","Data":"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"}
Nov 01 23:01:24.459033 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.458955    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5" event={"ID":"1d84ff32-5ab1-4588-b7de-a97ae8f70385","Type":"ContainerDied","Data":"1e222b54192b4bc10ac146fe189bb8454c89c3880e71db9b6c92ee97994fcfb4"}
Nov 01 23:01:24.459033 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.458986    5031 scope.go:122] "RemoveContainer" containerID="e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"
Nov 01 23:01:24.469680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.469651    5031 scope.go:122] "RemoveContainer" containerID="794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"
Nov 01 23:01:24.480142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.480119    5031 scope.go:122] "RemoveContainer" containerID="e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"
Nov 01 23:01:24.480487 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:01:24.480466    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd\": container with ID starting with e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd not found: ID does not exist" containerID="e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"
Nov 01 23:01:24.480560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.480497    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd"} err="failed to get container status \"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd\": rpc error: code = NotFound desc = could not find container \"e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd\": container with ID starting with e236e0e54788c73075e302675142dd60386c99497ac3a3964bbac8cf45ea47dd not found: ID does not exist"
Nov 01 23:01:24.480560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.480517    5031 scope.go:122] "RemoveContainer" containerID="794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"
Nov 01 23:01:24.480793 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:01:24.480772    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8\": container with ID starting with 794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8 not found: ID does not exist" containerID="794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"
Nov 01 23:01:24.480857 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.480803    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8"} err="failed to get container status \"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8\": rpc error: code = NotFound desc = could not find container \"794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8\": container with ID starting with 794219efaaac8e0edd737c041f9406156adcd90dc981dad9a138e177f934d7a8 not found: ID does not exist"
Nov 01 23:01:24.664315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.664279    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 23:01:24.710574 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:24.710472    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bd7ff8-openshift-default-6d58ddb979-tctl5"]
Nov 01 23:01:26.174284 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:26.174251    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" path="/var/lib/kubelet/pods/1d84ff32-5ab1-4588-b7de-a97ae8f70385/volumes"
Nov 01 23:01:40.626794 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:01:40.626756    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:01:56.879892 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.879856    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:01:56.880433 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880419    5031 cpu_manager.go:414] "containerMap: removing container" podUID="29557792-57fd-4908-8b2a-04dc5672b492" containerName="payload-pre-processing"
Nov 01 23:01:56.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880434    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29557792-57fd-4908-8b2a-04dc5672b492" containerName="payload-pre-processing"
Nov 01 23:01:56.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880463    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880468    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880482    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.880506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880487    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.880736 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880607    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="29557792-57fd-4908-8b2a-04dc5672b492" containerName="payload-pre-processing"
Nov 01 23:01:56.880736 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880622    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.880736 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.880631    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d84ff32-5ab1-4588-b7de-a97ae8f70385" containerName="istio-proxy"
Nov 01 23:01:56.884753 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.884730    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:56.933283 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.933246    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-55b58e-openshift-default-dockercfg-m25dq\""
Nov 01 23:01:57.011118 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:56.979824    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwsk\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.011470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.011434    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.011999 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.011915    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.012473 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.012340    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.013629 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.013604    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.013957 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.013903    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.014178 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.014148    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.014349 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.014334    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.014438 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.014425    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.014665 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.009477    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:01:57.116050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116004    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwsk\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116075    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116102    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116238    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116460 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116286    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116460 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116430    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116600 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116459    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116600 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116588    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116714 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116627    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116714 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116637    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116714 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116667    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116891 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116867    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.116954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.116907    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.117299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.117277    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.118738 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.118721    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.118891 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.118873    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.173082 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.172967    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.173360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.173326    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwsk\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\") pod \"e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") " pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.209951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.209918    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:57.656235 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.656199    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:01:57.662420 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:01:57.662383    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790c8db7_f00d_4ee9_9686_5b16f2c10d5e.slice/crio-b66cb64c53450af5a85a4554cbf09a4daa8c1d2d5880c0dd3b93e48f8c01c279 WatchSource:0}: Error finding container b66cb64c53450af5a85a4554cbf09a4daa8c1d2d5880c0dd3b93e48f8c01c279: Status 404 returned error can't find the container with id b66cb64c53450af5a85a4554cbf09a4daa8c1d2d5880c0dd3b93e48f8c01c279
Nov 01 23:01:57.665057 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.665023    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:01:57.665125 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.665112    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:01:57.665161 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.665140    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:01:57.687786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:57.687754    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerStarted","Data":"b66cb64c53450af5a85a4554cbf09a4daa8c1d2d5880c0dd3b93e48f8c01c279"}
Nov 01 23:01:58.695761 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:58.695720    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerStarted","Data":"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"}
Nov 01 23:01:58.811904 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:58.811844    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podStartSLOduration=2.811825871 podStartE2EDuration="2.811825871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:58.811303271 +0000 UTC m=+2272.767343116" watchObservedRunningTime="2026-11-01 23:01:58.811825871 +0000 UTC m=+2272.767865714"
Nov 01 23:01:59.210835 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:59.210801    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:59.216826 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:59.216796    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:59.702196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:59.702163    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:01:59.703292 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:01:59.703271    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:02:04.075812 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:04.075775    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 23:02:09.187471 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.187425    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 23:02:09.188013 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.187680    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" podUID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" containerName="maas-api" containerID="cri-o://f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b" gracePeriod=30
Nov 01 23:02:09.488729 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.488696    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 23:02:09.588929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.588894    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\") pod \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") "
Nov 01 23:02:09.589115 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.588979    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") pod \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") "
Nov 01 23:02:09.589115 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.589010    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\") pod \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") "
Nov 01 23:02:09.589115 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.589069    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\" (UniqueName: \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\") pod \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\" (UID: \"cac4d6d1-1ede-40ae-b07b-ec3681fe9020\") "
Nov 01 23:02:09.589413 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.589384    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca" pod "cac4d6d1-1ede-40ae-b07b-ec3681fe9020" (UID: "cac4d6d1-1ede-40ae-b07b-ec3681fe9020"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:09.589880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.589750    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:09.591385 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.591354    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls" pod "cac4d6d1-1ede-40ae-b07b-ec3681fe9020" (UID: "cac4d6d1-1ede-40ae-b07b-ec3681fe9020"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:09.591487 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.591394    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs" pod "cac4d6d1-1ede-40ae-b07b-ec3681fe9020" (UID: "cac4d6d1-1ede-40ae-b07b-ec3681fe9020"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:09.591487 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.591460    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv" pod "cac4d6d1-1ede-40ae-b07b-ec3681fe9020" (UID: "cac4d6d1-1ede-40ae-b07b-ec3681fe9020"). InnerVolumeSpecName "kube-api-access-mmcbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:09.691416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.691369    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:09.691416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.691407    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:09.691416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.691418    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcbv\" (UniqueName: \"kubernetes.io/projected/cac4d6d1-1ede-40ae-b07b-ec3681fe9020-kube-api-access-mmcbv\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:09.769688 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.769602    5031 generic.go:356] "Generic (PLEG): container finished" podID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" containerID="f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b" exitCode=0
Nov 01 23:02:09.769688 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.769670    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"
Nov 01 23:02:09.769890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.769679    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" event={"ID":"cac4d6d1-1ede-40ae-b07b-ec3681fe9020","Type":"ContainerDied","Data":"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"}
Nov 01 23:02:09.769890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.769734    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr" event={"ID":"cac4d6d1-1ede-40ae-b07b-ec3681fe9020","Type":"ContainerDied","Data":"71a05181d0c0be518df5dfe6f75ee9fd1839d102a02a07e813bdf08d62438e2c"}
Nov 01 23:02:09.769890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.769758    5031 scope.go:122] "RemoveContainer" containerID="f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"
Nov 01 23:02:09.783205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.783178    5031 scope.go:122] "RemoveContainer" containerID="f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"
Nov 01 23:02:09.783658 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:02:09.783609    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b\": container with ID starting with f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b not found: ID does not exist" containerID="f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"
Nov 01 23:02:09.783658 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.783660    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b"} err="failed to get container status \"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b\": rpc error: code = NotFound desc = could not find container \"f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b\": container with ID starting with f72d59f71d57daab5c3b25a8f1eda7d9c5aa30d0ec3afcf9c16a315f09991f4b not found: ID does not exist"
Nov 01 23:02:09.929581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.929499    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 23:02:09.934218 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:09.934190    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b8fd2-74fdfc7f59-dxgbr"]
Nov 01 23:02:10.172486 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:10.172441    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" path="/var/lib/kubelet/pods/cac4d6d1-1ede-40ae-b07b-ec3681fe9020/volumes"
Nov 01 23:02:10.626971 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:02:10.626936    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:02:19.097776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.097725    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main" containerID="cri-o://72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e" gracePeriod=45
Nov 01 23:02:19.365447 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.365371    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 23:02:19.487985 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.487951    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.488175 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488052    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.488175 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488155    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\" (UniqueName: \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.488307 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488205    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.488307 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488252    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.488307 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488253    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.488478 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.488383    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\") pod \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\" (UID: \"1ebb0115-9339-49d9-8ae3-cf0cf0012000\") "
Nov 01 23:02:19.489094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.489022    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.489590 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.489533    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.491718 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.491680    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:19.491718 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.491702    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "kube-api-access-2664l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:19.491912 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.491783    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.493026 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.492981    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir" pod "1ebb0115-9339-49d9-8ae3-cf0cf0012000" (UID: "1ebb0115-9339-49d9-8ae3-cf0cf0012000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.590447 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.590407    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.590447 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.590448    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2664l\" (UniqueName: \"kubernetes.io/projected/1ebb0115-9339-49d9-8ae3-cf0cf0012000-kube-api-access-2664l\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.590686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.590466    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.590686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.590481    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.590686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.590495    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebb0115-9339-49d9-8ae3-cf0cf0012000-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.870157 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.870123    5031 generic.go:356] "Generic (PLEG): container finished" podID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerID="72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e" exitCode=0
Nov 01 23:02:19.870377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.870170    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"
Nov 01 23:02:19.870377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.870202    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" event={"ID":"1ebb0115-9339-49d9-8ae3-cf0cf0012000","Type":"ContainerDied","Data":"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"}
Nov 01 23:02:19.870377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.870256    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf" event={"ID":"1ebb0115-9339-49d9-8ae3-cf0cf0012000","Type":"ContainerDied","Data":"0f5e7e75c28d81593b2e18683202a37feb24631f8837d904f015eecae4c5996f"}
Nov 01 23:02:19.870377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.870279    5031 scope.go:122] "RemoveContainer" containerID="72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"
Nov 01 23:02:19.881116 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.881091    5031 scope.go:122] "RemoveContainer" containerID="72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"
Nov 01 23:02:19.881436 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:02:19.881415    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e\": container with ID starting with 72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e not found: ID does not exist" containerID="72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"
Nov 01 23:02:19.881499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:19.881445    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e"} err="failed to get container status \"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e\": rpc error: code = NotFound desc = could not find container \"72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e\": container with ID starting with 72eb455d3e7fdbb16c6507798cff688ddd1415f13ed20e0017646c0578822b5e not found: ID does not exist"
Nov 01 23:02:20.025851 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:20.025814    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 23:02:20.071996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:20.071957    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/simulator-w5-d6101b-kserve-65db569bfb-zchqf"]
Nov 01 23:02:20.188708 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:02:20.188613    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" path="/var/lib/kubelet/pods/1ebb0115-9339-49d9-8ae3-cf0cf0012000/volumes"
Nov 01 23:02:40.627285 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:02:40.627247    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:03:02.469866 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:02.469827    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 23:03:02.516149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:02.516107    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 23:03:10.629513 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:03:10.629471    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:03:17.492031 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.491907    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main" containerID="cri-o://1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa" gracePeriod=45
Nov 01 23:03:17.541831 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.541785    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main" containerID="cri-o://1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137" gracePeriod=45
Nov 01 23:03:17.850484 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.850452    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 23:03:17.854056 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.854030    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 23:03:17.910148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910108    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.910350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910162    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\" (UniqueName: \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910193    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910212    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.910350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910324    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910347    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910387    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910461    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910514    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910593    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\") pod \"10f89fda-9319-49db-90c8-a91abd4318d1\" (UID: \"10f89fda-9319-49db-90c8-a91abd4318d1\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910522    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910628    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910639    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.910754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910664    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\" (UniqueName: \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\") pod \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\" (UID: \"a0e01ef8-e44c-41c2-8e4c-76b753c8c410\") "
Nov 01 23:03:17.911358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.910815    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.911358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911091    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.911480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911370    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.911562 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911491    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:17.911562 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911516    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:17.911562 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911530    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:17.911756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911562    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:17.911756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.911689    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.913210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913180    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:17.913210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913200    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.913402 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913240    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "kube-api-access-mczwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:17.913402 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913384    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:03:17.913624 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913602    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs" pod "10f89fda-9319-49db-90c8-a91abd4318d1" (UID: "10f89fda-9319-49db-90c8-a91abd4318d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:17.913860 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:17.913837    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx" pod "a0e01ef8-e44c-41c2-8e4c-76b753c8c410" (UID: "a0e01ef8-e44c-41c2-8e4c-76b753c8c410"). InnerVolumeSpecName "kube-api-access-s88xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:18.012840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012777    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.012840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012814    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88xx\" (UniqueName: \"kubernetes.io/projected/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-kube-api-access-s88xx\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.012840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012829    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.012840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012843    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczwq\" (UniqueName: \"kubernetes.io/projected/10f89fda-9319-49db-90c8-a91abd4318d1-kube-api-access-mczwq\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.013166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012857    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.013166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012871    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f89fda-9319-49db-90c8-a91abd4318d1-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.013166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012886    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e01ef8-e44c-41c2-8e4c-76b753c8c410-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.013166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.012898    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f89fda-9319-49db-90c8-a91abd4318d1-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.314857 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.314814    5031 generic.go:356] "Generic (PLEG): container finished" podID="10f89fda-9319-49db-90c8-a91abd4318d1" containerID="1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa" exitCode=0
Nov 01 23:03:18.315045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.314866    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"
Nov 01 23:03:18.315045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.314898    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" event={"ID":"10f89fda-9319-49db-90c8-a91abd4318d1","Type":"ContainerDied","Data":"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"}
Nov 01 23:03:18.315045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.314944    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr" event={"ID":"10f89fda-9319-49db-90c8-a91abd4318d1","Type":"ContainerDied","Data":"69a1945c9ed9e62278d06c718d92f9293d980327c2afff0de773afede690bdde"}
Nov 01 23:03:18.315045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.314961    5031 scope.go:122] "RemoveContainer" containerID="1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"
Nov 01 23:03:18.316557 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.316515    5031 generic.go:356] "Generic (PLEG): container finished" podID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerID="1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137" exitCode=0
Nov 01 23:03:18.316673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.316573    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" event={"ID":"a0e01ef8-e44c-41c2-8e4c-76b753c8c410","Type":"ContainerDied","Data":"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"}
Nov 01 23:03:18.316673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.316594    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"
Nov 01 23:03:18.316778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.316601    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8" event={"ID":"a0e01ef8-e44c-41c2-8e4c-76b753c8c410","Type":"ContainerDied","Data":"71d8b9fe2bf415b678c77b43952cb4900f5c5bf3a24323d9769140f1bacaaf15"}
Nov 01 23:03:18.325411 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.325392    5031 scope.go:122] "RemoveContainer" containerID="1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"
Nov 01 23:03:18.325809 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:03:18.325789    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa\": container with ID starting with 1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa not found: ID does not exist" containerID="1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"
Nov 01 23:03:18.325875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.325819    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa"} err="failed to get container status \"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa\": rpc error: code = NotFound desc = could not find container \"1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa\": container with ID starting with 1ee2eb07f47c021fb8d42ba9835200689c6c41e1710bfc8af5351365c16588fa not found: ID does not exist"
Nov 01 23:03:18.325875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.325838    5031 scope.go:122] "RemoveContainer" containerID="1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"
Nov 01 23:03:18.335100 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.335074    5031 scope.go:122] "RemoveContainer" containerID="1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"
Nov 01 23:03:18.335453 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:03:18.335429    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137\": container with ID starting with 1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137 not found: ID does not exist" containerID="1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"
Nov 01 23:03:18.335524 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.335463    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137"} err="failed to get container status \"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137\": rpc error: code = NotFound desc = could not find container \"1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137\": container with ID starting with 1a454d54b101f58d4dd867631a0dce50a4f04e79eb2e1a8a1e6a933d65009137 not found: ID does not exist"
Nov 01 23:03:18.468682 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.468596    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 23:03:18.514828 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.514794    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/premium-w3-e71d55-kserve-5f949b8d74-g4wv8"]
Nov 01 23:03:18.616533 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.616502    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 23:03:18.619255 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:18.619222    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-e71d55/distinct-2-w3-e71d55-kserve-5d6bf4d4d6-922lr"]
Nov 01 23:03:20.172164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:20.172120    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" path="/var/lib/kubelet/pods/10f89fda-9319-49db-90c8-a91abd4318d1/volumes"
Nov 01 23:03:20.172793 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:20.172771    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" path="/var/lib/kubelet/pods/a0e01ef8-e44c-41c2-8e4c-76b753c8c410/volumes"
Nov 01 23:03:40.626712 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:03:40.626673    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:03:50.754657 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:50.754615    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 23:03:50.755171 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:50.754848    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" podUID="26552a01-8249-404d-92cb-113439596f52" containerName="maas-api" containerID="cri-o://cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56" gracePeriod=30
Nov 01 23:03:51.041250 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.041211    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 23:03:51.097861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.097829    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\" (UniqueName: \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\") pod \"26552a01-8249-404d-92cb-113439596f52\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") "
Nov 01 23:03:51.098058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.097929    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") pod \"26552a01-8249-404d-92cb-113439596f52\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") "
Nov 01 23:03:51.098058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.097964    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\") pod \"26552a01-8249-404d-92cb-113439596f52\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") "
Nov 01 23:03:51.098058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.098003    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\") pod \"26552a01-8249-404d-92cb-113439596f52\" (UID: \"26552a01-8249-404d-92cb-113439596f52\") "
Nov 01 23:03:51.098377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.098334    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca" pod "26552a01-8249-404d-92cb-113439596f52" (UID: "26552a01-8249-404d-92cb-113439596f52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:51.098849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.098816    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26552a01-8249-404d-92cb-113439596f52-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:51.100389 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.100358    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls" pod "26552a01-8249-404d-92cb-113439596f52" (UID: "26552a01-8249-404d-92cb-113439596f52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:51.100510 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.100388    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs" pod "26552a01-8249-404d-92cb-113439596f52" (UID: "26552a01-8249-404d-92cb-113439596f52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:51.100510 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.100441    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n" pod "26552a01-8249-404d-92cb-113439596f52" (UID: "26552a01-8249-404d-92cb-113439596f52"). InnerVolumeSpecName "kube-api-access-j5s9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:51.200582 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.200516    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5s9n\" (UniqueName: \"kubernetes.io/projected/26552a01-8249-404d-92cb-113439596f52-kube-api-access-j5s9n\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:51.200759 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.200587    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:51.200759 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.200603    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26552a01-8249-404d-92cb-113439596f52-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:51.546721 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.546681    5031 generic.go:356] "Generic (PLEG): container finished" podID="26552a01-8249-404d-92cb-113439596f52" containerID="cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56" exitCode=0
Nov 01 23:03:51.546911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.546754    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"
Nov 01 23:03:51.546911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.546765    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" event={"ID":"26552a01-8249-404d-92cb-113439596f52","Type":"ContainerDied","Data":"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"}
Nov 01 23:03:51.546911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.546848    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4" event={"ID":"26552a01-8249-404d-92cb-113439596f52","Type":"ContainerDied","Data":"137ffec9046522e5dcbc3de155afb548c4738708b03eb1d6a8103b6e21e602bc"}
Nov 01 23:03:51.546911 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.546872    5031 scope.go:122] "RemoveContainer" containerID="cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"
Nov 01 23:03:51.558069 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.558045    5031 scope.go:122] "RemoveContainer" containerID="cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"
Nov 01 23:03:51.558422 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:03:51.558401    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56\": container with ID starting with cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56 not found: ID does not exist" containerID="cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"
Nov 01 23:03:51.558510 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.558431    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56"} err="failed to get container status \"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56\": rpc error: code = NotFound desc = could not find container \"cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56\": container with ID starting with cb9f66e0b91de3218d25e0518a315316d5b5dd04ea3edae64939ad52b6684d56 not found: ID does not exist"
Nov 01 23:03:51.721467 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.721429    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 23:03:51.761997 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:51.757236    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a97162-5b58f8f64b-sf9t4"]
Nov 01 23:03:52.171387 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:52.171348    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26552a01-8249-404d-92cb-113439596f52" path="/var/lib/kubelet/pods/26552a01-8249-404d-92cb-113439596f52/volumes"
Nov 01 23:03:59.870323 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:59.870290    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 23:03:59.870775 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:03:59.870523    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" containerID="cri-o://2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b" gracePeriod=30
Nov 01 23:04:04.075586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:04.075525    5031 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:04:04.075998 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:04.075659    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:04:05.154069 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.154041    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 23:04:05.192407 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192370    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192416    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192453    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192479    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192795 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192613    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192795 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192662    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192795 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192691    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192795    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:05.192965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192803    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.192965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192836    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:05.192965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192945    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\") pod \"f485be50-f54d-40c3-9644-08b2744e302c\" (UID: \"f485be50-f54d-40c3-9644-08b2744e302c\") "
Nov 01 23:04:05.193169 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.192975    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:05.193169 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193084    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:05.193437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193174    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:05.193745 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193724    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.193833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193749    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.193833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193762    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f485be50-f54d-40c3-9644-08b2744e302c-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.193833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193777    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.193833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.193791    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.195081 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.195053    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:05.195211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.195144    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "kube-api-access-z27xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:05.195632 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.195612    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:05.196163 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.196150    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy" pod "f485be50-f54d-40c3-9644-08b2744e302c" (UID: "f485be50-f54d-40c3-9644-08b2744e302c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:05.295051 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.295015    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27xp\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-kube-api-access-z27xp\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.295051 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.295046    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f485be50-f54d-40c3-9644-08b2744e302c-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.295051 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.295056    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f485be50-f54d-40c3-9644-08b2744e302c-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.295287 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.295065    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f485be50-f54d-40c3-9644-08b2744e302c-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.640935 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.640894    5031 generic.go:356] "Generic (PLEG): container finished" podID="f485be50-f54d-40c3-9644-08b2744e302c" containerID="2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b" exitCode=0
Nov 01 23:04:05.641159 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.640964    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerDied","Data":"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"}
Nov 01 23:04:05.641159 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.641033    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9" event={"ID":"f485be50-f54d-40c3-9644-08b2744e302c","Type":"ContainerDied","Data":"e568e7e80b01c4594b382b0c2819084358399530d796695a58c6335f9fcfc54a"}
Nov 01 23:04:05.641159 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.641063    5031 scope.go:122] "RemoveContainer" containerID="2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"
Nov 01 23:04:05.641491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.641464    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"
Nov 01 23:04:05.653461 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.653357    5031 scope.go:122] "RemoveContainer" containerID="41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"
Nov 01 23:04:05.664117 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.664096    5031 scope.go:122] "RemoveContainer" containerID="2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"
Nov 01 23:04:05.664441 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:05.664420    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b\": container with ID starting with 2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b not found: ID does not exist" containerID="2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"
Nov 01 23:04:05.664486 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.664453    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b"} err="failed to get container status \"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b\": rpc error: code = NotFound desc = could not find container \"2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b\": container with ID starting with 2106eabb29445c8038fd7a1ac1863bc64873bc84c2a979fff33f61bd62b3999b not found: ID does not exist"
Nov 01 23:04:05.664486 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.664474    5031 scope.go:122] "RemoveContainer" containerID="41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"
Nov 01 23:04:05.664794 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:05.664770    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6\": container with ID starting with 41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6 not found: ID does not exist" containerID="41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"
Nov 01 23:04:05.664872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.664805    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6"} err="failed to get container status \"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6\": rpc error: code = NotFound desc = could not find container \"41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6\": container with ID starting with 41ea9ae1d67ad4f6a9b1a5a92dddde47530c9eb4f022ab421b940ad57078afb6 not found: ID does not exist"
Nov 01 23:04:05.796239 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.796196    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 23:04:05.842367 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:05.842330    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a97162-openshift-default-7fcfff4c79-mbwq9"]
Nov 01 23:04:06.173182 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:06.173152    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f485be50-f54d-40c3-9644-08b2744e302c" path="/var/lib/kubelet/pods/f485be50-f54d-40c3-9644-08b2744e302c/volumes"
Nov 01 23:04:10.632233 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:10.632193    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:04:12.286852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.286811    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 23:04:12.332783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.332748    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 23:04:12.332972 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.332949    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" podUID="54bef2ed-5d69-413b-8ab2-c6d127797e70" containerName="maas-api" containerID="cri-o://7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c" gracePeriod=30
Nov 01 23:04:12.396343 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.396292    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 23:04:12.643792 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.643761    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 23:04:12.711046 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.711002    5031 generic.go:356] "Generic (PLEG): container finished" podID="54bef2ed-5d69-413b-8ab2-c6d127797e70" containerID="7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c" exitCode=0
Nov 01 23:04:12.711336 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.711129    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" event={"ID":"54bef2ed-5d69-413b-8ab2-c6d127797e70","Type":"ContainerDied","Data":"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"}
Nov 01 23:04:12.711336 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.711212    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28" event={"ID":"54bef2ed-5d69-413b-8ab2-c6d127797e70","Type":"ContainerDied","Data":"8f6959e3290b484c2c33c396fd1f78b6a55b45a3dca388cbcde8d8e1e8b29e8a"}
Nov 01 23:04:12.711336 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.711256    5031 scope.go:122] "RemoveContainer" containerID="7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"
Nov 01 23:04:12.711568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.711465    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"
Nov 01 23:04:12.718153 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.718103    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\") pod \"54bef2ed-5d69-413b-8ab2-c6d127797e70\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") "
Nov 01 23:04:12.718324 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.718290    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") pod \"54bef2ed-5d69-413b-8ab2-c6d127797e70\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") "
Nov 01 23:04:12.718456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.718438    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\" (UniqueName: \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\") pod \"54bef2ed-5d69-413b-8ab2-c6d127797e70\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") "
Nov 01 23:04:12.718529 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.718513    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\") pod \"54bef2ed-5d69-413b-8ab2-c6d127797e70\" (UID: \"54bef2ed-5d69-413b-8ab2-c6d127797e70\") "
Nov 01 23:04:12.719313 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.719284    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca" pod "54bef2ed-5d69-413b-8ab2-c6d127797e70" (UID: "54bef2ed-5d69-413b-8ab2-c6d127797e70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:12.720083 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.720056    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54bef2ed-5d69-413b-8ab2-c6d127797e70-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:12.723376 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.723336    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p" pod "54bef2ed-5d69-413b-8ab2-c6d127797e70" (UID: "54bef2ed-5d69-413b-8ab2-c6d127797e70"). InnerVolumeSpecName "kube-api-access-xj42p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:12.756573 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.756495    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs" pod "54bef2ed-5d69-413b-8ab2-c6d127797e70" (UID: "54bef2ed-5d69-413b-8ab2-c6d127797e70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:12.773350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.764488    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls" pod "54bef2ed-5d69-413b-8ab2-c6d127797e70" (UID: "54bef2ed-5d69-413b-8ab2-c6d127797e70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:12.773350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.766373    5031 scope.go:122] "RemoveContainer" containerID="7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"
Nov 01 23:04:12.773350 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:12.766962    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c\": container with ID starting with 7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c not found: ID does not exist" containerID="7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"
Nov 01 23:04:12.773350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.766999    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c"} err="failed to get container status \"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c\": rpc error: code = NotFound desc = could not find container \"7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c\": container with ID starting with 7ec5668808bef4fec681dfb5e1e16f85123ec5229cd019a56c9e9d8f43ea321c not found: ID does not exist"
Nov 01 23:04:12.821345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.821306    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:12.821345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.821346    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54bef2ed-5d69-413b-8ab2-c6d127797e70-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:12.821642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:12.821363    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj42p\" (UniqueName: \"kubernetes.io/projected/54bef2ed-5d69-413b-8ab2-c6d127797e70-kube-api-access-xj42p\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:13.126994 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:13.126956    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 23:04:13.131434 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:13.131400    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-2b0866-68d7dd8977-h8z28"]
Nov 01 23:04:14.171528 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:14.171486    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bef2ed-5d69-413b-8ab2-c6d127797e70" path="/var/lib/kubelet/pods/54bef2ed-5d69-413b-8ab2-c6d127797e70/volumes"
Nov 01 23:04:15.743734 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:15.743698    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:04:15.744215 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:15.743773    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:04:16.741329 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.741298    5031 generic.go:356] "Generic (PLEG): container finished" podID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerID="09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725" exitCode=0
Nov 01 23:04:16.741594 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.741371    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerDied","Data":"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"}
Nov 01 23:04:16.741848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.741827    5031 scope.go:122] "RemoveContainer" containerID="09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"
Nov 01 23:04:16.743584 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.743530    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:16.743691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.743632    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:16.743691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:16.743678    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:17.288673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.288618    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" podUID="5888fbd6-9378-4282-ad03-6c472c5b67c5" containerName="payload-processing" containerID="cri-o://5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca" gracePeriod=25
Nov 01 23:04:17.397661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.397620    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" podUID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" containerName="payload-pre-processing" containerID="cri-o://2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340" gracePeriod=25
Nov 01 23:04:17.638236 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.636152    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 23:04:17.743168 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.743137    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 23:04:17.748455 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.748426    5031 generic.go:356] "Generic (PLEG): container finished" podID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" containerID="2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340" exitCode=0
Nov 01 23:04:17.748647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.748486    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"
Nov 01 23:04:17.748647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.748509    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" event={"ID":"01eba096-753a-48c3-8f7f-6ec6f1a99e35","Type":"ContainerDied","Data":"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"}
Nov 01 23:04:17.748647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.748593    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k" event={"ID":"01eba096-753a-48c3-8f7f-6ec6f1a99e35","Type":"ContainerDied","Data":"f8d47745d7a7029937c2d380adb11c6985b08e4082387119bb895dcd52e47bed"}
Nov 01 23:04:17.748647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.748632    5031 scope.go:122] "RemoveContainer" containerID="2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"
Nov 01 23:04:17.750034 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.750015    5031 generic.go:356] "Generic (PLEG): container finished" podID="5888fbd6-9378-4282-ad03-6c472c5b67c5" containerID="5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca" exitCode=0
Nov 01 23:04:17.750155 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.750084    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"
Nov 01 23:04:17.750155 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.750143    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" event={"ID":"5888fbd6-9378-4282-ad03-6c472c5b67c5","Type":"ContainerDied","Data":"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"}
Nov 01 23:04:17.750256 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.750190    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd" event={"ID":"5888fbd6-9378-4282-ad03-6c472c5b67c5","Type":"ContainerDied","Data":"426d5162a632ff719cea6c7cdaea6ef1fac68871464cfc1b6d9437545dc5a33b"}
Nov 01 23:04:17.752969 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.752945    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerStarted","Data":"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"}
Nov 01 23:04:17.792565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.787987    5031 scope.go:122] "RemoveContainer" containerID="2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"
Nov 01 23:04:17.792565 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:17.789858    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340\": container with ID starting with 2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340 not found: ID does not exist" containerID="2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"
Nov 01 23:04:17.792565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.789899    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340"} err="failed to get container status \"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340\": rpc error: code = NotFound desc = could not find container \"2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340\": container with ID starting with 2eef3c0b9f6fa76052bdc7ee3df9bb3d2137bb18fa91a37ac286eab53b2ba340 not found: ID does not exist"
Nov 01 23:04:17.792565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.789925    5031 scope.go:122] "RemoveContainer" containerID="5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"
Nov 01 23:04:17.793222 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.793193    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\") pod \"5888fbd6-9378-4282-ad03-6c472c5b67c5\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") "
Nov 01 23:04:17.793374 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.793360    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\" (UniqueName: \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\") pod \"5888fbd6-9378-4282-ad03-6c472c5b67c5\" (UID: \"5888fbd6-9378-4282-ad03-6c472c5b67c5\") "
Nov 01 23:04:17.794716 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.794595    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume" pod "5888fbd6-9378-4282-ad03-6c472c5b67c5" (UID: "5888fbd6-9378-4282-ad03-6c472c5b67c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:17.801197 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.801164    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw" pod "5888fbd6-9378-4282-ad03-6c472c5b67c5" (UID: "5888fbd6-9378-4282-ad03-6c472c5b67c5"). InnerVolumeSpecName "kube-api-access-qbstw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:17.802164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.802084    5031 scope.go:122] "RemoveContainer" containerID="5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"
Nov 01 23:04:17.802628 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:17.802531    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca\": container with ID starting with 5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca not found: ID does not exist" containerID="5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"
Nov 01 23:04:17.802628 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.802605    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca"} err="failed to get container status \"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca\": rpc error: code = NotFound desc = could not find container \"5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca\": container with ID starting with 5e7f5ae77deec76c3e73cd09c0a72ada4a7ab14873599e33c4ddf9960350f4ca not found: ID does not exist"
Nov 01 23:04:17.895338 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.895252    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\") pod \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") "
Nov 01 23:04:17.895492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.895370    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\" (UniqueName: \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\") pod \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\" (UID: \"01eba096-753a-48c3-8f7f-6ec6f1a99e35\") "
Nov 01 23:04:17.904778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.895964    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume" pod "01eba096-753a-48c3-8f7f-6ec6f1a99e35" (UID: "01eba096-753a-48c3-8f7f-6ec6f1a99e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:17.904778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.896674    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01eba096-753a-48c3-8f7f-6ec6f1a99e35-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:17.904778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.896710    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5888fbd6-9378-4282-ad03-6c472c5b67c5-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:17.904778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.896727    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbstw\" (UniqueName: \"kubernetes.io/projected/5888fbd6-9378-4282-ad03-6c472c5b67c5-kube-api-access-qbstw\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:17.904778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.898753    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c" pod "01eba096-753a-48c3-8f7f-6ec6f1a99e35" (UID: "01eba096-753a-48c3-8f7f-6ec6f1a99e35"). InnerVolumeSpecName "kube-api-access-8xg4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:17.998128 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:17.998084    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xg4c\" (UniqueName: \"kubernetes.io/projected/01eba096-753a-48c3-8f7f-6ec6f1a99e35-kube-api-access-8xg4c\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:18.198781 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.198698    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 23:04:18.202122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.202094    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2b0866-5d7c78fbcf-nrt5k"]
Nov 01 23:04:18.210576 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.210505    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:04:18.210878 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.210854    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 23:04:18.210959 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.210907    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 23:04:18.324172 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.324130    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 23:04:18.342006 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.341974    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-2b0866-75f8b5fc5-xt9sd"]
Nov 01 23:04:18.679224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.679182    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 23:04:18.679476 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:18.679445    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" podUID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" containerName="maas-api" containerID="cri-o://d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff" gracePeriod=30
Nov 01 23:04:19.013709 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.013675    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 23:04:19.024604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.024577    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\" (UniqueName: \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\") pod \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") "
Nov 01 23:04:19.024713 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.024672    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\") pod \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") "
Nov 01 23:04:19.024713 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.024704    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\") pod \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") "
Nov 01 23:04:19.024812 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.024796    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") pod \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\" (UID: \"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15\") "
Nov 01 23:04:19.025095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.025066    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca" pod "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" (UID: "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:19.025342 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.025323    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:19.027089 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.027062    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls" pod "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" (UID: "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:19.027531 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.027507    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs" pod "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" (UID: "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:19.027628 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.027531    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w" pod "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" (UID: "43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15"). InnerVolumeSpecName "kube-api-access-z9q5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:19.126190 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.126157    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:19.126190 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.126186    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:19.126190 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.126196    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9q5w\" (UniqueName: \"kubernetes.io/projected/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15-kube-api-access-z9q5w\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:19.210882 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.210842    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 23:04:19.211096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.210919    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 23:04:19.793299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.793261    5031 generic.go:356] "Generic (PLEG): container finished" podID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" containerID="d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff" exitCode=0
Nov 01 23:04:19.793299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.793309    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"
Nov 01 23:04:19.793905 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.793354    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" event={"ID":"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15","Type":"ContainerDied","Data":"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"}
Nov 01 23:04:19.793905 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.793406    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d" event={"ID":"43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15","Type":"ContainerDied","Data":"30d13d405b454120593c7a978044123e4dc2a0e9c65c589fab4df44c1b176c6d"}
Nov 01 23:04:19.793905 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.793429    5031 scope.go:122] "RemoveContainer" containerID="d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"
Nov 01 23:04:19.803411 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.803388    5031 scope.go:122] "RemoveContainer" containerID="d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"
Nov 01 23:04:19.803779 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:19.803753    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff\": container with ID starting with d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff not found: ID does not exist" containerID="d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"
Nov 01 23:04:19.803846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.803785    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff"} err="failed to get container status \"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff\": rpc error: code = NotFound desc = could not find container \"d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff\": container with ID starting with d30341cabe1ba0d39fa7547210bb5fd20cc875f416987e9422da7f2b7b6e4aff not found: ID does not exist"
Nov 01 23:04:19.901043 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.901005    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 23:04:19.901250 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.901225    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerName="maas-api" containerID="cri-o://f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead" gracePeriod=30
Nov 01 23:04:19.946286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.946252    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 23:04:19.998625 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:19.995194    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e71d55-5679b44f57-2vj5d"]
Nov 01 23:04:20.091088 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.090992    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.39:8443/health\": dial tcp 10.132.0.39:8443: connect: connection refused"
Nov 01 23:04:20.210591 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.207823    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" path="/var/lib/kubelet/pods/01eba096-753a-48c3-8f7f-6ec6f1a99e35/volumes"
Nov 01 23:04:20.210591 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.208268    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" path="/var/lib/kubelet/pods/43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15/volumes"
Nov 01 23:04:20.210591 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.208706    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5888fbd6-9378-4282-ad03-6c472c5b67c5" path="/var/lib/kubelet/pods/5888fbd6-9378-4282-ad03-6c472c5b67c5/volumes"
Nov 01 23:04:20.210591 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.210278    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 23:04:20.210591 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.210324    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 23:04:20.345076 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.344999    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 23:04:20.485963 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.485928    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") pod \"d8a585c4-897e-41d0-b122-19f8b56682e0\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") "
Nov 01 23:04:20.488226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.486297    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\") pod \"d8a585c4-897e-41d0-b122-19f8b56682e0\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") "
Nov 01 23:04:20.488226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.487143    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\") pod \"d8a585c4-897e-41d0-b122-19f8b56682e0\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") "
Nov 01 23:04:20.488226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.487253    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\" (UniqueName: \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\") pod \"d8a585c4-897e-41d0-b122-19f8b56682e0\" (UID: \"d8a585c4-897e-41d0-b122-19f8b56682e0\") "
Nov 01 23:04:20.488226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.487565    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca" pod "d8a585c4-897e-41d0-b122-19f8b56682e0" (UID: "d8a585c4-897e-41d0-b122-19f8b56682e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:20.488226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.488026    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a585c4-897e-41d0-b122-19f8b56682e0-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:20.490491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.488984    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls" pod "d8a585c4-897e-41d0-b122-19f8b56682e0" (UID: "d8a585c4-897e-41d0-b122-19f8b56682e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:20.490491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.489150    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs" pod "d8a585c4-897e-41d0-b122-19f8b56682e0" (UID: "d8a585c4-897e-41d0-b122-19f8b56682e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:20.491750 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.491475    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh" pod "d8a585c4-897e-41d0-b122-19f8b56682e0" (UID: "d8a585c4-897e-41d0-b122-19f8b56682e0"). InnerVolumeSpecName "kube-api-access-bwwmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:20.589470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.589431    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwwmh\" (UniqueName: \"kubernetes.io/projected/d8a585c4-897e-41d0-b122-19f8b56682e0-kube-api-access-bwwmh\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:20.589470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.589464    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:20.589470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.589473    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8a585c4-897e-41d0-b122-19f8b56682e0-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:20.803991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.803949    5031 generic.go:356] "Generic (PLEG): container finished" podID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerID="f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead" exitCode=0
Nov 01 23:04:20.804504 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.804057    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"
Nov 01 23:04:20.804504 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.804107    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" event={"ID":"d8a585c4-897e-41d0-b122-19f8b56682e0","Type":"ContainerDied","Data":"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"}
Nov 01 23:04:20.804504 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.804212    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg" event={"ID":"d8a585c4-897e-41d0-b122-19f8b56682e0","Type":"ContainerDied","Data":"21461e6cebbe69398713aeb0798997b88157bd0357c11eadc984855366f1d6ad"}
Nov 01 23:04:20.804504 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.804239    5031 scope.go:122] "RemoveContainer" containerID="f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"
Nov 01 23:04:20.816879 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.816846    5031 scope.go:122] "RemoveContainer" containerID="f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"
Nov 01 23:04:20.817198 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:20.817178    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead\": container with ID starting with f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead not found: ID does not exist" containerID="f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"
Nov 01 23:04:20.817264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:20.817208    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead"} err="failed to get container status \"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead\": rpc error: code = NotFound desc = could not find container \"f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead\": container with ID starting with f523803d3a68e023c2103e830be1601ab7192daefa4c32e934ac991a15a02ead not found: ID does not exist"
Nov 01 23:04:21.014565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:21.009593    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 23:04:21.014565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:21.010038    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-d6101b-dddb85b4d-rx7qg"]
Nov 01 23:04:21.210371 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:21.210284    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 23:04:21.210371 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:21.210341    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 23:04:22.185967 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:22.185930    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" path="/var/lib/kubelet/pods/d8a585c4-897e-41d0-b122-19f8b56682e0/volumes"
Nov 01 23:04:22.210327 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:22.210290    5031 patch_prober.go:29] interesting pod/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 23:04:22.210514 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:22.210342    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 23:04:23.214995 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:23.214963    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:04:23.215470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:23.215398    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:04:23.216253 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:23.216236    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:04:27.674722 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:27.674681    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 23:04:27.675223 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:27.674974    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy" containerID="cri-o://ee9a3c19dc87d505ac6903b96ff49eff7dcf6c12027eaa4988195aca5035ab31" gracePeriod=30
Nov 01 23:04:29.483774 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:29.483728    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 23:04:29.484370 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:29.483995    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" containerID="cri-o://eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9" gracePeriod=30
Nov 01 23:04:31.038797 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.038758    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:04:31.039286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039273    5031 cpu_manager.go:414] "containerMap: removing container" podUID="26552a01-8249-404d-92cb-113439596f52" containerName="maas-api"
Nov 01 23:04:31.039339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039287    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26552a01-8249-404d-92cb-113439596f52" containerName="maas-api"
Nov 01 23:04:31.039339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039302    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerName="maas-api"
Nov 01 23:04:31.039339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039307    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerName="maas-api"
Nov 01 23:04:31.039339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039324    5031 cpu_manager.go:414] "containerMap: removing container" podUID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" containerName="maas-api"
Nov 01 23:04:31.039339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039329    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" containerName="maas-api"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039346    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039350    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039361    5031 cpu_manager.go:414] "containerMap: removing container" podUID="5888fbd6-9378-4282-ad03-6c472c5b67c5" containerName="payload-processing"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039366    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5888fbd6-9378-4282-ad03-6c472c5b67c5" containerName="payload-processing"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039375    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039379    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039386    5031 cpu_manager.go:414] "containerMap: removing container" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039390    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039398    5031 cpu_manager.go:414] "containerMap: removing container" podUID="54bef2ed-5d69-413b-8ab2-c6d127797e70" containerName="maas-api"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039402    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bef2ed-5d69-413b-8ab2-c6d127797e70" containerName="maas-api"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039410    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" containerName="maas-api"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039415    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" containerName="maas-api"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039426    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039430    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039438    5031 cpu_manager.go:414] "containerMap: removing container" podUID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" containerName="payload-pre-processing"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039443    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" containerName="payload-pre-processing"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039452    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main"
Nov 01 23:04:31.039492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039456    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039532    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="26552a01-8249-404d-92cb-113439596f52" containerName="maas-api"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039561    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a585c4-897e-41d0-b122-19f8b56682e0" containerName="maas-api"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039572    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a3621a-b3bf-4ed8-a4f6-caf25dd8bc15" containerName="maas-api"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039589    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039608    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="5888fbd6-9378-4282-ad03-6c472c5b67c5" containerName="payload-processing"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039617    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485be50-f54d-40c3-9644-08b2744e302c" containerName="istio-proxy"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039625    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f89fda-9319-49db-90c8-a91abd4318d1" containerName="main"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039634    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bef2ed-5d69-413b-8ab2-c6d127797e70" containerName="maas-api"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039646    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac4d6d1-1ede-40ae-b07b-ec3681fe9020" containerName="maas-api"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039664    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e01ef8-e44c-41c2-8e4c-76b753c8c410" containerName="main"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039669    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eba096-753a-48c3-8f7f-6ec6f1a99e35" containerName="payload-pre-processing"
Nov 01 23:04:31.040245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.039675    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebb0115-9339-49d9-8ae3-cf0cf0012000" containerName="main"
Nov 01 23:04:31.043204 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.043176    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.108609 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.096709    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3eb631-openshift-default-dockercfg-jg8bv\""
Nov 01 23:04:31.142788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.142746    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.142810    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143020    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143074    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143190    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143222    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfs4x\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143341    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143383    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.143507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.143489    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.146664 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.146634    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:04:31.244275 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244245    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244286    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244321    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244338    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfs4x\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244371    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244389    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244427    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244445    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.244886 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.244551    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.245067 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.245043    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.245150 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.245121    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.245150 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.245142    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.245295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.245173    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.245295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.245213    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.247157 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.247132    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.247283 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.247270    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.297733 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.297701    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.297897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.297764    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfs4x\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\") pod \"e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") " pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.361318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.361279    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:31.800050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.800006    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:04:31.805092 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:04:31.805049    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e3c20d_d3a2_46d3_98c6_56ca0017a485.slice/crio-a534c597aeaf3bdf65e3fa99cc3fc947b133022df03f9555c03fae26b80dbde1 WatchSource:0}: Error finding container a534c597aeaf3bdf65e3fa99cc3fc947b133022df03f9555c03fae26b80dbde1: Status 404 returned error can't find the container with id a534c597aeaf3bdf65e3fa99cc3fc947b133022df03f9555c03fae26b80dbde1
Nov 01 23:04:31.807963 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.807923    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:31.808070 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.807996    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:31.808070 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.808025    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:04:31.831615 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.831581    5031 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:04:31.831770 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.831636    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:04:31.908130 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.908093    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerStarted","Data":"4817f55b047f2b88f68a9e6ce5b1efcf3a8a78422d159ac18eed64707abb9831"}
Nov 01 23:04:31.908322 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:31.908151    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerStarted","Data":"a534c597aeaf3bdf65e3fa99cc3fc947b133022df03f9555c03fae26b80dbde1"}
Nov 01 23:04:32.080723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.080618    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podStartSLOduration=2.080600383 podStartE2EDuration="2.080600383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:32.078579543 +0000 UTC m=+2426.034619387" watchObservedRunningTime="2026-11-01 23:04:32.080600383 +0000 UTC m=+2426.036640227"
Nov 01 23:04:32.283164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.283121    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:04:32.287266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.287240    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.336353 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.336275    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4af44d-serving-cert\""
Nov 01 23:04:32.348368 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.348330    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:04:32.355481 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.355435    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4m6\" (UniqueName: \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.355672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.355494    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.355741 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.355716    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.355788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.355770    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.362030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.361996    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:32.368203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.368172    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:32.457401 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.457359    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4m6\" (UniqueName: \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.457620 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.457427    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.457620 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.457560    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.457620 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.457609    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.457895 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:32.457876    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-serving-cert: secret "maas-api-e2e-inf-b-4af44d-serving-cert" not found
Nov 01 23:04:32.457965 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:32.457953    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls podName:359a82d1-a799-4f64-ac60-2ddd85732fc3 nodeName:}" failed. No retries permitted until 2026-11-01 23:04:32.957931392 +0000 UTC m=+2426.913971429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls") pod "maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" (UID: "359a82d1-a799-4f64-ac60-2ddd85732fc3") : secret "maas-api-e2e-inf-b-4af44d-serving-cert" not found
Nov 01 23:04:32.458255 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.458203    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.460756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.460731    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.514878 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.514845    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4m6\" (UniqueName: \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.917242 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.917208    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerID="ee9a3c19dc87d505ac6903b96ff49eff7dcf6c12027eaa4988195aca5035ab31" exitCode=0
Nov 01 23:04:32.917448 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.917281    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" event={"ID":"e4c41ea7-baab-49f6-801d-6043182ee02e","Type":"ContainerDied","Data":"ee9a3c19dc87d505ac6903b96ff49eff7dcf6c12027eaa4988195aca5035ab31"}
Nov 01 23:04:32.917674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.917651    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:32.918688 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.918665    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:04:32.952859 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.952832    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 23:04:32.966811 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.966781    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.966957 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.966856    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967013    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967050    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967116 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967101    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967185 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967164    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:32.967239 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967239 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967210    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967351 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967280    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967351 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967298    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:32.967351 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967333    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\") pod \"e4c41ea7-baab-49f6-801d-6043182ee02e\" (UID: \"e4c41ea7-baab-49f6-801d-6043182ee02e\") "
Nov 01 23:04:32.967517 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967416    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:32.967759 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967674    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:32.967902 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.967867    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:32.968069 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:32.968599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968190    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c41ea7-baab-49f6-801d-6043182ee02e-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:32.968599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968212    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:32.968599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968227    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:32.968599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968240    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:32.968599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.968253    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:32.969964 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.969939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:32.970085 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.969969    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "kube-api-access-lflzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:32.970381 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.970357    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:32.970381 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.970370    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token" pod "e4c41ea7-baab-49f6-801d-6043182ee02e" (UID: "e4c41ea7-baab-49f6-801d-6043182ee02e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:32.970759 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:32.970739    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") pod \"maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:33.069598 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.069560    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.069598 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.069603    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.069833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.069617    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflzx\" (UniqueName: \"kubernetes.io/projected/e4c41ea7-baab-49f6-801d-6043182ee02e-kube-api-access-lflzx\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.069833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.069632    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c41ea7-baab-49f6-801d-6043182ee02e-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:33.203362 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.203262    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:33.579580 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.579517    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:04:33.586100 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:04:33.586059    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359a82d1_a799_4f64_ac60_2ddd85732fc3.slice/crio-8c912bad77b55e1317205c934cb6fdb6b870af48ac5b95d5015963ae738070dc WatchSource:0}: Error finding container 8c912bad77b55e1317205c934cb6fdb6b870af48ac5b95d5015963ae738070dc: Status 404 returned error can't find the container with id 8c912bad77b55e1317205c934cb6fdb6b870af48ac5b95d5015963ae738070dc
Nov 01 23:04:33.587687 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.587665    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:04:33.925367 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.925271    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" event={"ID":"359a82d1-a799-4f64-ac60-2ddd85732fc3","Type":"ContainerStarted","Data":"8c912bad77b55e1317205c934cb6fdb6b870af48ac5b95d5015963ae738070dc"}
Nov 01 23:04:33.926775 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.926751    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"
Nov 01 23:04:33.926919 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.926808    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss" event={"ID":"e4c41ea7-baab-49f6-801d-6043182ee02e","Type":"ContainerDied","Data":"113cf42ddd07354d36716e93a935b9afdb009bb1fbf4f148bf275062adc52e08"}
Nov 01 23:04:33.926919 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:33.926867    5031 scope.go:122] "RemoveContainer" containerID="ee9a3c19dc87d505ac6903b96ff49eff7dcf6c12027eaa4988195aca5035ab31"
Nov 01 23:04:34.125829 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.125791    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 23:04:34.172332 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.172290    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-qf2ss"]
Nov 01 23:04:34.757263 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.757230    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 23:04:34.792210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792172    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792380 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792240    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792380 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792275    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792469 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792393    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792469 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792414    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792469 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792440    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792488    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792572    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792665    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\") pod \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\" (UID: \"3b11edd3-a9a7-4607-bd7a-9615671ef7df\") "
Nov 01 23:04:34.792862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792785    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:34.792862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792768    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.792862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.792827    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.793055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793029    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.793281 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793033    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.793352 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793328    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.793408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793352    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.793408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793368    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.793408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793382    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.793408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.793395    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.794818 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.794789    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.794965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.794790    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "kube-api-access-4n5s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.794965 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.794893    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.795054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.794964    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo" pod "3b11edd3-a9a7-4607-bd7a-9615671ef7df" (UID: "3b11edd3-a9a7-4607-bd7a-9615671ef7df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:34.894399 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.894360    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5s2\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-kube-api-access-4n5s2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.894399 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.894397    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.894645 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.894411    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.894645 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.894424    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11edd3-a9a7-4607-bd7a-9615671ef7df-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.937474 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.937373    5031 generic.go:356] "Generic (PLEG): container finished" podID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerID="eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9" exitCode=0
Nov 01 23:04:34.937474 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.937430    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"
Nov 01 23:04:34.937474 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.937457    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerDied","Data":"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"}
Nov 01 23:04:34.937838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.937516    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr" event={"ID":"3b11edd3-a9a7-4607-bd7a-9615671ef7df","Type":"ContainerDied","Data":"d4371ccd7243a7ce92e29d0bc41657cfb3b57bedfed4e8d4ea6465ed272fb7ce"}
Nov 01 23:04:34.937838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.937578    5031 scope.go:122] "RemoveContainer" containerID="eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"
Nov 01 23:04:34.941245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.940145    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" event={"ID":"359a82d1-a799-4f64-ac60-2ddd85732fc3","Type":"ContainerStarted","Data":"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"}
Nov 01 23:04:34.941245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.940266    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:34.950484 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.950462    5031 scope.go:122] "RemoveContainer" containerID="a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"
Nov 01 23:04:34.963142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.963098    5031 scope.go:122] "RemoveContainer" containerID="eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"
Nov 01 23:04:34.964224 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:34.963521    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9\": container with ID starting with eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9 not found: ID does not exist" containerID="eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"
Nov 01 23:04:34.964224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.963592    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9"} err="failed to get container status \"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9\": rpc error: code = NotFound desc = could not find container \"eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9\": container with ID starting with eefc8476191484c693f4949ccf167b750499d0766dba750c8f04dce335f20fd9 not found: ID does not exist"
Nov 01 23:04:34.964224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.963618    5031 scope.go:122] "RemoveContainer" containerID="a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"
Nov 01 23:04:34.964224 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:34.964214    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6\": container with ID starting with a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6 not found: ID does not exist" containerID="a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"
Nov 01 23:04:34.964933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:34.964253    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6"} err="failed to get container status \"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6\": rpc error: code = NotFound desc = could not find container \"a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6\": container with ID starting with a3588b243e2204e05f64433e2e3694467f3dcfcfed8e00c5a3c96477b17d7da6 not found: ID does not exist"
Nov 01 23:04:35.099999 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:35.099943    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" podStartSLOduration=2.301981992 podStartE2EDuration="3.099927893s" totalImagesPullingTime="797.945901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:35.097374379 +0000 UTC m=+2429.053414224" watchObservedRunningTime="2026-11-01 23:04:35.099927893 +0000 UTC m=+2429.055967739"
Nov 01 23:04:35.202237 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:35.202162    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 23:04:35.207142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:35.207106    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-gdghr"]
Nov 01 23:04:36.170880 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:36.170843    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" path="/var/lib/kubelet/pods/3b11edd3-a9a7-4607-bd7a-9615671ef7df/volumes"
Nov 01 23:04:36.171413 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:36.171397    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" path="/var/lib/kubelet/pods/e4c41ea7-baab-49f6-801d-6043182ee02e/volumes"
Nov 01 23:04:40.627635 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:40.627588    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:04:40.985412 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:40.964334    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:04:42.681191 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681149    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:04:42.681715 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681698    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:42.681776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681716    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:42.681776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681752    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy"
Nov 01 23:04:42.681776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681757    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy"
Nov 01 23:04:42.681875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681834    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:42.681875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681848    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:42.681875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.681857    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c41ea7-baab-49f6-801d-6043182ee02e" containerName="istio-proxy"
Nov 01 23:04:42.686478 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.686458    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.703326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.703299    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:04:42.734858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.734823    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cf46c7-serving-cert\""
Nov 01 23:04:42.801166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.801135    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.801354 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.801298    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.801402 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.801373    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq2c2\" (UniqueName: \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.801443 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.801408    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.902511 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.902457    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.902798 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.902582    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq2c2\" (UniqueName: \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.902798 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.902638    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.902798 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.902714    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.903311 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:42.903282    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-serving-cert: secret "maas-api-e2e-inf-b-cf46c7-serving-cert" not found
Nov 01 23:04:42.903455 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:04:42.903385    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls podName:a6aed3a7-7bfd-4c62-a636-df97ceb3211a nodeName:}" failed. No retries permitted until 2026-11-01 23:04:43.403365317 +0000 UTC m=+2437.359405154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls") pod "maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" (UID: "a6aed3a7-7bfd-4c62-a636-df97ceb3211a") : secret "maas-api-e2e-inf-b-cf46c7-serving-cert" not found
Nov 01 23:04:42.904222 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.904197    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.906633 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.906608    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:42.958149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:42.958075    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq2c2\" (UniqueName: \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:43.408864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.408828    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:43.411483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.411452    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:43.601045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.601014    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:43.819167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.819132    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:04:43.819898 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.819876    5031 cpu_manager.go:414] "containerMap: removing container" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:43.819898 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.819899    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b11edd3-a9a7-4607-bd7a-9615671ef7df" containerName="istio-proxy"
Nov 01 23:04:43.824704 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.824678    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:43.877188 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.877153    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3eb631\""
Nov 01 23:04:43.877459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.877398    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3eb631-dockercfg-h28tp\""
Nov 01 23:04:43.926510 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.926389    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:43.926737 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.926672    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wks6\" (UniqueName: \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:43.930163 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.930110    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:04:43.984841 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.984795    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:04:43.989854 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:43.989824    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.006506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.005410    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:04:44.011375 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:04:44.011341    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aed3a7_7bfd_4c62_a636_df97ceb3211a.slice/crio-0bf6dc3058db5a8a4aef994314bcdab3d15c55081ac3c1af6afadfbcff87b7b9 WatchSource:0}: Error finding container 0bf6dc3058db5a8a4aef994314bcdab3d15c55081ac3c1af6afadfbcff87b7b9: Status 404 returned error can't find the container with id 0bf6dc3058db5a8a4aef994314bcdab3d15c55081ac3c1af6afadfbcff87b7b9
Nov 01 23:04:44.027408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.027371    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wks6\" (UniqueName: \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:44.027655 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.027464    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:44.028213 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.028189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:44.029827 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.029797    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" event={"ID":"a6aed3a7-7bfd-4c62-a636-df97ceb3211a","Type":"ContainerStarted","Data":"0bf6dc3058db5a8a4aef994314bcdab3d15c55081ac3c1af6afadfbcff87b7b9"}
Nov 01 23:04:44.035790 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.035760    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:04:44.081865 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.081824    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wks6\" (UniqueName: \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\") pod \"payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:44.128966 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.128929    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddzbj\" (UniqueName: \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.129146 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.129100    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.139707 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.139668    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:44.231091 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.231053    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddzbj\" (UniqueName: \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.231299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.231227    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.232252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.232225    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.289102 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.289070    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddzbj\" (UniqueName: \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\") pod \"payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") " pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.306987 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.306947    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:44.513566 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.513451    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:04:44.519659 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:04:44.519617    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb8f7ba_375f_480d_8e7b_b8238cde6934.slice/crio-d93fdc81d85cbdd11e9fea0ad3a1ee541fd78be4752efa5f0ed650671001a080 WatchSource:0}: Error finding container d93fdc81d85cbdd11e9fea0ad3a1ee541fd78be4752efa5f0ed650671001a080: Status 404 returned error can't find the container with id d93fdc81d85cbdd11e9fea0ad3a1ee541fd78be4752efa5f0ed650671001a080
Nov 01 23:04:44.679587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:44.679534    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:04:44.684449 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:04:44.684414    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612ecb6b_f262_4723_b71a_3c77ffc45938.slice/crio-de2f0e56f967b88f31156372e39655c7b9e824f0440ff6009d4e5a47addf02a8 WatchSource:0}: Error finding container de2f0e56f967b88f31156372e39655c7b9e824f0440ff6009d4e5a47addf02a8: Status 404 returned error can't find the container with id de2f0e56f967b88f31156372e39655c7b9e824f0440ff6009d4e5a47addf02a8
Nov 01 23:04:45.037598 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.037530    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" event={"ID":"a6aed3a7-7bfd-4c62-a636-df97ceb3211a","Type":"ContainerStarted","Data":"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"}
Nov 01 23:04:45.038053 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.037629    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:45.039860 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.039825    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" event={"ID":"612ecb6b-f262-4723-b71a-3c77ffc45938","Type":"ContainerStarted","Data":"5bc8fbd6e6239b53d6be940a413a3af96ec182812a9bf0ad6a1ac6042638ec7b"}
Nov 01 23:04:45.040025 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.039890    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" event={"ID":"612ecb6b-f262-4723-b71a-3c77ffc45938","Type":"ContainerStarted","Data":"de2f0e56f967b88f31156372e39655c7b9e824f0440ff6009d4e5a47addf02a8"}
Nov 01 23:04:45.040025 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.039922    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:45.041523 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.041498    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" event={"ID":"6fb8f7ba-375f-480d-8e7b-b8238cde6934","Type":"ContainerStarted","Data":"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"}
Nov 01 23:04:45.041670 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.041593    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:04:45.041670 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.041625    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" event={"ID":"6fb8f7ba-375f-480d-8e7b-b8238cde6934","Type":"ContainerStarted","Data":"d93fdc81d85cbdd11e9fea0ad3a1ee541fd78be4752efa5f0ed650671001a080"}
Nov 01 23:04:45.147270 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.147146    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" podStartSLOduration=2.29383511 podStartE2EDuration="3.14712845s" totalImagesPullingTime="853.29334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:45.145064973 +0000 UTC m=+2439.101104820" watchObservedRunningTime="2026-11-01 23:04:45.14712845 +0000 UTC m=+2439.103168295"
Nov 01 23:04:45.281493 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.281411    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" podStartSLOduration=2.281382449 podStartE2EDuration="2.281382449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:45.265502561 +0000 UTC m=+2439.221542405" watchObservedRunningTime="2026-11-01 23:04:45.281382449 +0000 UTC m=+2439.237422294"
Nov 01 23:04:45.393595 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:45.393247    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" podStartSLOduration=2.393222469 podStartE2EDuration="2.393222469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:45.384823541 +0000 UTC m=+2439.340863386" watchObservedRunningTime="2026-11-01 23:04:45.393222469 +0000 UTC m=+2439.349262313"
Nov 01 23:04:51.054059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:51.054028    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:04:56.050673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:56.050638    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:04:56.051183 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:04:56.050726    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:05:08.140762 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:05:08.140729    5031 scope.go:122] "RemoveContainer" containerID="9f82af32a6385931d7f1e7c4e955e1fe94464ec7e2f999be6e312f623d7040b1"
Nov 01 23:05:10.627329 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:05:10.627289    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:05:40.626848 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:05:40.626808    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:06:03.855111 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:03.853733    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": read tcp 10.132.0.2:48926->10.132.0.106:15021: read: connection reset by peer" start-of-body=
Nov 01 23:06:03.855111 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:03.853838    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": read tcp 10.132.0.2:48926->10.132.0.106:15021: read: connection reset by peer"
Nov 01 23:06:04.555776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.555741    5031 generic.go:356] "Generic (PLEG): container finished" podID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerID="4817f55b047f2b88f68a9e6ce5b1efcf3a8a78422d159ac18eed64707abb9831" exitCode=0
Nov 01 23:06:04.555953 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.555799    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerDied","Data":"4817f55b047f2b88f68a9e6ce5b1efcf3a8a78422d159ac18eed64707abb9831"}
Nov 01 23:06:04.556203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.556190    5031 scope.go:122] "RemoveContainer" containerID="4817f55b047f2b88f68a9e6ce5b1efcf3a8a78422d159ac18eed64707abb9831"
Nov 01 23:06:04.557836 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.557804    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:06:04.557914 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.557878    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:06:04.557914 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:04.557911    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:06:05.593587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:05.593531    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerStarted","Data":"f5301b85468f8cd212c89bf0a8e5be374190f7a7444b49821b1d4bb90cea90a0"}
Nov 01 23:06:06.369028 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:06.368985    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:06:06.369408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:06.369384    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 23:06:06.369629 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:06.369600    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 23:06:07.368717 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:07.368677    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 23:06:07.369370 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:07.369336    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 23:06:08.374489 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:08.373992    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 23:06:08.374489 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:08.374076    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 23:06:09.363664 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:09.363617    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 23:06:09.363979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:09.363951    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 23:06:10.369045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:10.368998    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 23:06:10.369747 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:10.369703    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 23:06:10.634773 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:06:10.634682    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:06:11.366265 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:11.366232    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:06:11.366599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:11.366566    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:06:11.367387 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:06:11.367371    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:06:40.626386 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:06:40.626303    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:07:08.208974 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:08.208939    5031 scope.go:122] "RemoveContainer" containerID="f6147dc216ab220a5fe8abe251f82e6efa7cc3944cc58b1b132870b83e9c276e"
Nov 01 23:07:10.627048 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:07:10.627009    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:07:39.918344 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:39.918308    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 23:07:39.919057 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:39.918567    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" containerID="cri-o://ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded" gracePeriod=30
Nov 01 23:07:40.627280 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:07:40.627243    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:07:41.208753 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:41.208712    5031 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:07:41.209183 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:41.208777    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:07:45.201316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.201288    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 23:07:45.206873 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.206840    5031 generic.go:356] "Generic (PLEG): container finished" podID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerID="ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded" exitCode=0
Nov 01 23:07:45.207072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.206916    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"
Nov 01 23:07:45.207072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.206932    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerDied","Data":"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"}
Nov 01 23:07:45.207072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.207033    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz" event={"ID":"0a203e67-a35d-4644-a6ac-ba1d1de22534","Type":"ContainerDied","Data":"fb5933304b816f57085dfd438163020335b1026e81116e78bd585fe38ec53e73"}
Nov 01 23:07:45.207072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.207069    5031 scope.go:122] "RemoveContainer" containerID="ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"
Nov 01 23:07:45.216957 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.216934    5031 scope.go:122] "RemoveContainer" containerID="c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"
Nov 01 23:07:45.228613 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.228587    5031 scope.go:122] "RemoveContainer" containerID="ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"
Nov 01 23:07:45.228965 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:07:45.228943    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded\": container with ID starting with ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded not found: ID does not exist" containerID="ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"
Nov 01 23:07:45.229057 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.228975    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded"} err="failed to get container status \"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded\": rpc error: code = NotFound desc = could not find container \"ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded\": container with ID starting with ee0321e5ba854fd2b6b8bc94fb2bf6d6593e4b98ee8a56173754f7387588fded not found: ID does not exist"
Nov 01 23:07:45.229057 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.228995    5031 scope.go:122] "RemoveContainer" containerID="c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"
Nov 01 23:07:45.229254 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:07:45.229239    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b\": container with ID starting with c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b not found: ID does not exist" containerID="c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"
Nov 01 23:07:45.229308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.229258    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b"} err="failed to get container status \"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b\": rpc error: code = NotFound desc = could not find container \"c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b\": container with ID starting with c21d254f36b30d76ea6145a92fb816d93d2409d17b763d2ef59bcca17918749b not found: ID does not exist"
Nov 01 23:07:45.308264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308228    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308264 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308268    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308301    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308331    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308376    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308415    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308826 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308521    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.308826 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308581    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:45.308826 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308675    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\") pod \"0a203e67-a35d-4644-a6ac-ba1d1de22534\" (UID: \"0a203e67-a35d-4644-a6ac-ba1d1de22534\") "
Nov 01 23:07:45.309001 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308882    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:45.309001 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308918    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:45.309001 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.308947    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:45.309172 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309089    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:45.309485 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309460    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.309654 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309487    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.309654 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309503    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.309654 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309519    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.309654 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.309533    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a203e67-a35d-4644-a6ac-ba1d1de22534-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.310859 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.310827    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:07:45.310977 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.310927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "kube-api-access-bl87z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:45.310977 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.310950    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:45.311164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.311144    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy" pod "0a203e67-a35d-4644-a6ac-ba1d1de22534" (UID: "0a203e67-a35d-4644-a6ac-ba1d1de22534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:07:45.411317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.411235    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.411317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.411274    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.411317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.411288    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a203e67-a35d-4644-a6ac-ba1d1de22534-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.411317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.411302    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl87z\" (UniqueName: \"kubernetes.io/projected/0a203e67-a35d-4644-a6ac-ba1d1de22534-kube-api-access-bl87z\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:45.622592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.622552    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 23:07:45.626024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:45.625995    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-wbkbz"]
Nov 01 23:07:46.171330 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:07:46.171294    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" path="/var/lib/kubelet/pods/0a203e67-a35d-4644-a6ac-ba1d1de22534/volumes"
Nov 01 23:08:10.627165 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:08:10.627128    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:08:40.627121 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:08:40.627082    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:09:00.163179 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:00.163143    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:09:00.163754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:00.163433    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy" containerID="cri-o://4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b" gracePeriod=30
Nov 01 23:09:05.439351 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.439279    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:09:05.488900 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.488868    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489077 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.488929    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489077 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.488976    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489077 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489009    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489134    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489175    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489212    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:05.489342 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489314    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489394 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489367    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:05.489452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489377    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489394    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:05.489452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489439    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\") pod \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\" (UID: \"790c8db7-f00d-4ee9-9686-5b16f2c10d5e\") "
Nov 01 23:09:05.489655 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489634    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:05.489726 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.489703    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:05.490148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.490060    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.490148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.490080    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.490148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.490090    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.490148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.490099    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.490148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.490113    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.491503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.491474    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "kube-api-access-vpwsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:05.491776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.491758    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:05.491982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.491961    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:09:05.491982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.491966    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token" pod "790c8db7-f00d-4ee9-9686-5b16f2c10d5e" (UID: "790c8db7-f00d-4ee9-9686-5b16f2c10d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:05.591603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.591561    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.591603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.591605    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.591873 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.591621    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpwsk\" (UniqueName: \"kubernetes.io/projected/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-kube-api-access-vpwsk\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.591873 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.591639    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790c8db7-f00d-4ee9-9686-5b16f2c10d5e-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:05.680116 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.680076    5031 generic.go:356] "Generic (PLEG): container finished" podID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerID="4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b" exitCode=0
Nov 01 23:09:05.680294 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.680123    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerDied","Data":"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"}
Nov 01 23:09:05.680294 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.680131    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"
Nov 01 23:09:05.680294 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.680166    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx" event={"ID":"790c8db7-f00d-4ee9-9686-5b16f2c10d5e","Type":"ContainerDied","Data":"b66cb64c53450af5a85a4554cbf09a4daa8c1d2d5880c0dd3b93e48f8c01c279"}
Nov 01 23:09:05.680294 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.680195    5031 scope.go:122] "RemoveContainer" containerID="4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"
Nov 01 23:09:05.691516 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.691493    5031 scope.go:122] "RemoveContainer" containerID="09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"
Nov 01 23:09:05.703977 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.703952    5031 scope.go:122] "RemoveContainer" containerID="4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"
Nov 01 23:09:05.704327 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:05.704304    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b\": container with ID starting with 4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b not found: ID does not exist" containerID="4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"
Nov 01 23:09:05.704467 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.704340    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b"} err="failed to get container status \"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b\": rpc error: code = NotFound desc = could not find container \"4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b\": container with ID starting with 4587e943a9f8163ae05c88e8c1e68c0b12d76d511ff7b74dd0618d603df8540b not found: ID does not exist"
Nov 01 23:09:05.704467 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.704372    5031 scope.go:122] "RemoveContainer" containerID="09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"
Nov 01 23:09:05.704695 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:05.704675    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725\": container with ID starting with 09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725 not found: ID does not exist" containerID="09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"
Nov 01 23:09:05.704762 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.704701    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725"} err="failed to get container status \"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725\": rpc error: code = NotFound desc = could not find container \"09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725\": container with ID starting with 09df027c033c724059a487447cb5432455fcc4cf63f644beafadec4e86850725 not found: ID does not exist"
Nov 01 23:09:05.834569 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.834503    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:09:05.840205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:05.840176    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-55b58e-openshift-default-5b79ff477c-xfcnx"]
Nov 01 23:09:06.171062 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:06.171029    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" path="/var/lib/kubelet/pods/790c8db7-f00d-4ee9-9686-5b16f2c10d5e/volumes"
Nov 01 23:09:10.626869 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:10.626827    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:09:12.495937 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:12.495898    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:09:12.664852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:12.664817    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:09:17.496283 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.496160    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" podUID="612ecb6b-f262-4723-b71a-3c77ffc45938" containerName="payload-processing" containerID="cri-o://5bc8fbd6e6239b53d6be940a413a3af96ec182812a9bf0ad6a1ac6042638ec7b" gracePeriod=25
Nov 01 23:09:17.665798 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.665750    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" podUID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" containerName="payload-pre-processing" containerID="cri-o://2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb" gracePeriod=25
Nov 01 23:09:17.754897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.754818    5031 generic.go:356] "Generic (PLEG): container finished" podID="612ecb6b-f262-4723-b71a-3c77ffc45938" containerID="5bc8fbd6e6239b53d6be940a413a3af96ec182812a9bf0ad6a1ac6042638ec7b" exitCode=0
Nov 01 23:09:17.755050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.754889    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" event={"ID":"612ecb6b-f262-4723-b71a-3c77ffc45938","Type":"ContainerDied","Data":"5bc8fbd6e6239b53d6be940a413a3af96ec182812a9bf0ad6a1ac6042638ec7b"}
Nov 01 23:09:17.797587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.797562    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:09:17.921456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.921426    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:09:17.960749 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.960708    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\" (UniqueName: \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\") pod \"612ecb6b-f262-4723-b71a-3c77ffc45938\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") "
Nov 01 23:09:17.960926 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.960809    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\") pod \"612ecb6b-f262-4723-b71a-3c77ffc45938\" (UID: \"612ecb6b-f262-4723-b71a-3c77ffc45938\") "
Nov 01 23:09:17.961282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.961255    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume" pod "612ecb6b-f262-4723-b71a-3c77ffc45938" (UID: "612ecb6b-f262-4723-b71a-3c77ffc45938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:17.963198 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:17.963172    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj" pod "612ecb6b-f262-4723-b71a-3c77ffc45938" (UID: "612ecb6b-f262-4723-b71a-3c77ffc45938"). InnerVolumeSpecName "kube-api-access-ddzbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:18.062483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.062439    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\") pod \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") "
Nov 01 23:09:18.062705 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.062571    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\" (UniqueName: \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\") pod \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\" (UID: \"6fb8f7ba-375f-480d-8e7b-b8238cde6934\") "
Nov 01 23:09:18.062915 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.062891    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume" pod "6fb8f7ba-375f-480d-8e7b-b8238cde6934" (UID: "6fb8f7ba-375f-480d-8e7b-b8238cde6934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:18.063130 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.063112    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/612ecb6b-f262-4723-b71a-3c77ffc45938-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:18.063196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.063135    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb8f7ba-375f-480d-8e7b-b8238cde6934-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:18.063196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.063149    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddzbj\" (UniqueName: \"kubernetes.io/projected/612ecb6b-f262-4723-b71a-3c77ffc45938-kube-api-access-ddzbj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:18.064833 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.064811    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6" pod "6fb8f7ba-375f-480d-8e7b-b8238cde6934" (UID: "6fb8f7ba-375f-480d-8e7b-b8238cde6934"). InnerVolumeSpecName "kube-api-access-4wks6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:18.164866 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.164835    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wks6\" (UniqueName: \"kubernetes.io/projected/6fb8f7ba-375f-480d-8e7b-b8238cde6934-kube-api-access-4wks6\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:18.762018 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.761984    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"
Nov 01 23:09:18.762018 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.762005    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54" event={"ID":"612ecb6b-f262-4723-b71a-3c77ffc45938","Type":"ContainerDied","Data":"de2f0e56f967b88f31156372e39655c7b9e824f0440ff6009d4e5a47addf02a8"}
Nov 01 23:09:18.762609 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.762081    5031 scope.go:122] "RemoveContainer" containerID="5bc8fbd6e6239b53d6be940a413a3af96ec182812a9bf0ad6a1ac6042638ec7b"
Nov 01 23:09:18.763662 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.763623    5031 generic.go:356] "Generic (PLEG): container finished" podID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" containerID="2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb" exitCode=0
Nov 01 23:09:18.763763 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.763722    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"
Nov 01 23:09:18.763763 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.763745    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" event={"ID":"6fb8f7ba-375f-480d-8e7b-b8238cde6934","Type":"ContainerDied","Data":"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"}
Nov 01 23:09:18.763846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.763797    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v" event={"ID":"6fb8f7ba-375f-480d-8e7b-b8238cde6934","Type":"ContainerDied","Data":"d93fdc81d85cbdd11e9fea0ad3a1ee541fd78be4752efa5f0ed650671001a080"}
Nov 01 23:09:18.771508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.771484    5031 scope.go:122] "RemoveContainer" containerID="2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"
Nov 01 23:09:18.780415 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.780394    5031 scope.go:122] "RemoveContainer" containerID="2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"
Nov 01 23:09:18.780758 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:18.780739    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb\": container with ID starting with 2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb not found: ID does not exist" containerID="2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"
Nov 01 23:09:18.780820 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.780768    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb"} err="failed to get container status \"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb\": rpc error: code = NotFound desc = could not find container \"2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb\": container with ID starting with 2fffeae24c71159b056dcf268d951904a7bdb364e7727c949b9560994c3f73bb not found: ID does not exist"
Nov 01 23:09:18.916457 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.916424    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:09:18.962483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:18.962447    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3eb631-6b478cd87f-ktb2v"]
Nov 01 23:09:19.064997 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:19.064965    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:09:19.069643 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:19.069608    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3eb631-7d796dfd97-zql54"]
Nov 01 23:09:20.171797 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:20.171758    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612ecb6b-f262-4723-b71a-3c77ffc45938" path="/var/lib/kubelet/pods/612ecb6b-f262-4723-b71a-3c77ffc45938/volumes"
Nov 01 23:09:20.172362 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:20.172343    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" path="/var/lib/kubelet/pods/6fb8f7ba-375f-480d-8e7b-b8238cde6934/volumes"
Nov 01 23:09:40.626396 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:40.626354    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:09:46.664502 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.664465    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 23:09:46.665021 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.664729    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery" containerID="cri-o://7e508a471ca251ad42c931a12616e97cc4f9ef277625b799ee539badba75c8d0" gracePeriod=30
Nov 01 23:09:46.926058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.926025    5031 generic.go:356] "Generic (PLEG): container finished" podID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerID="7e508a471ca251ad42c931a12616e97cc4f9ef277625b799ee539badba75c8d0" exitCode=0
Nov 01 23:09:46.926211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.926105    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" event={"ID":"cabeed4b-35e8-4266-88c1-6bbec4c22cd8","Type":"ContainerDied","Data":"7e508a471ca251ad42c931a12616e97cc4f9ef277625b799ee539badba75c8d0"}
Nov 01 23:09:46.926211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.926167    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc" event={"ID":"cabeed4b-35e8-4266-88c1-6bbec4c22cd8","Type":"ContainerDied","Data":"93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9"}
Nov 01 23:09:46.926211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.926188    5031 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b771b837f801d92b72ff1a96d0cbc617aa9c9cb00d46c6bd76eabe9cc76ec9"
Nov 01 23:09:46.935530 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:46.935508    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 23:09:47.002100 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002069    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002309 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002145    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002309 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002169    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002309 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002209    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002309 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002275    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002532 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002344    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.002532 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.002394    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\") pod \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\" (UID: \"cabeed4b-35e8-4266-88c1-6bbec4c22cd8\") "
Nov 01 23:09:47.003272 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.003237    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:47.004615 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.004581    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:47.004743 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.004620    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:47.005052 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.005023    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:47.005126 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.005044    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:47.005210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.005195    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "kube-api-access-skdh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:47.005482 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.005467    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig" pod "cabeed4b-35e8-4266-88c1-6bbec4c22cd8" (UID: "cabeed4b-35e8-4266-88c1-6bbec4c22cd8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:47.104028 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.103988    5031 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-kubeconfig\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104028 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104029    5031 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-ca-configmap\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104233 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104041    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdh2\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-kube-api-access-skdh2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104233 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104051    5031 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-local-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104233 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104059    5031 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-cacerts\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104233 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104067    5031 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-csr-dns-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.104233 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.104075    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabeed4b-35e8-4266-88c1-6bbec4c22cd8-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:47.931927 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:47.931890    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"
Nov 01 23:09:48.131783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:48.131745    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 23:09:48.177930 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:48.177896    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vrjtc"]
Nov 01 23:09:50.171218 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:50.171184    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" path="/var/lib/kubelet/pods/cabeed4b-35e8-4266-88c1-6bbec4c22cd8/volumes"
Nov 01 23:09:53.082208 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:53.082169    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 23:09:53.082818 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:53.082411    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy" containerID="cri-o://0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725" gracePeriod=30
Nov 01 23:09:58.350406 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.350332    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 23:09:58.442859 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.442821    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.442881    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.442942    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.442975    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443038    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443075    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443104    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443132    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443213    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\") pod \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\" (UID: \"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49\") "
Nov 01 23:09:58.443492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443331    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:58.443492 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443367    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:58.443647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443614    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:58.443720 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443709    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:58.443813 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443791    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.443813 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443793    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:58.443813 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443813    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.444068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443829    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.444068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.443841    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.445534 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.445504    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:58.445725 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.445659    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:09:58.445824 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.445801    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:58.445881 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.445817    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x" pod "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" (UID: "a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49"). InnerVolumeSpecName "kube-api-access-npc5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:58.544874 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.544839    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.544874 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.544868    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npc5x\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-kube-api-access-npc5x\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.544874 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.544879    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.545191 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.544888    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.545191 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.544896    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:58.997842 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.997803    5031 generic.go:356] "Generic (PLEG): container finished" podID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerID="0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725" exitCode=0
Nov 01 23:09:58.998023 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.997855    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"
Nov 01 23:09:58.998023 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.997888    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" event={"ID":"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49","Type":"ContainerDied","Data":"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"}
Nov 01 23:09:58.998023 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.997947    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw" event={"ID":"a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49","Type":"ContainerDied","Data":"0e96df5a471ad974bb2f05a339329cd192244ac0d4ab06875fde3fdd5c76de62"}
Nov 01 23:09:58.998023 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:58.997976    5031 scope.go:122] "RemoveContainer" containerID="0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"
Nov 01 23:09:59.008507 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:59.008487    5031 scope.go:122] "RemoveContainer" containerID="0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"
Nov 01 23:09:59.008826 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:09:59.008807    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725\": container with ID starting with 0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725 not found: ID does not exist" containerID="0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"
Nov 01 23:09:59.008892 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:59.008839    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725"} err="failed to get container status \"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725\": rpc error: code = NotFound desc = could not find container \"0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725\": container with ID starting with 0105ce34381f6c2265ffa21db73028267585232a6b0926847a6b251c49502725 not found: ID does not exist"
Nov 01 23:09:59.153073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:59.153037    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 23:09:59.199383 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:09:59.199344    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-xvntw"]
Nov 01 23:10:00.170963 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:00.170931    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" path="/var/lib/kubelet/pods/a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49/volumes"
Nov 01 23:10:08.266055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:08.266017    5031 scope.go:122] "RemoveContainer" containerID="7e508a471ca251ad42c931a12616e97cc4f9ef277625b799ee539badba75c8d0"
Nov 01 23:10:10.626791 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:10:10.626750    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:10:40.626172 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:10:40.626132    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:10:42.930768 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:42.930727    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:10:43.047732 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:43.047697    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:10:47.931664 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:47.931561    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" podUID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" containerName="payload-processing" containerID="cri-o://94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30" gracePeriod=25
Nov 01 23:10:48.048646 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.048595    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing" containerID="cri-o://0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f" gracePeriod=25
Nov 01 23:10:48.226956 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.226927    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:10:48.280249 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.280213    5031 generic.go:356] "Generic (PLEG): container finished" podID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" containerID="94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30" exitCode=0
Nov 01 23:10:48.280459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.280266    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"
Nov 01 23:10:48.280459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.280278    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" event={"ID":"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea","Type":"ContainerDied","Data":"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"}
Nov 01 23:10:48.280459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.280359    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn" event={"ID":"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea","Type":"ContainerDied","Data":"b0a83e9455d2b5ab6d1f6bc6a4ff6c22e8725e6dd39dfeb59f569bbd16a9e42f"}
Nov 01 23:10:48.280459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.280389    5031 scope.go:122] "RemoveContainer" containerID="94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"
Nov 01 23:10:48.289513 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.289494    5031 scope.go:122] "RemoveContainer" containerID="94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"
Nov 01 23:10:48.289875 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:10:48.289853    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30\": container with ID starting with 94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30 not found: ID does not exist" containerID="94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"
Nov 01 23:10:48.289938 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.289885    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30"} err="failed to get container status \"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30\": rpc error: code = NotFound desc = could not find container \"94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30\": container with ID starting with 94a5efe6fcecc9d1a54142035eec009b9295ac193fb806ce99b8b69139e66c30 not found: ID does not exist"
Nov 01 23:10:48.331680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.331637    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\" (UniqueName: \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\") pod \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") "
Nov 01 23:10:48.331892 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.331751    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\") pod \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\" (UID: \"0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea\") "
Nov 01 23:10:48.332137 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.332110    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume" pod "0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" (UID: "0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:48.334024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.333995    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7" pod "0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" (UID: "0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea"). InnerVolumeSpecName "kube-api-access-mgjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:48.433616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.433569    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjj7\" (UniqueName: \"kubernetes.io/projected/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-kube-api-access-mgjj7\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:48.433616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.433608    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:48.697167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.697133    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:10:48.702826 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:48.702795    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-675d1d-59898b6955-d6rbn"]
Nov 01 23:10:50.171192 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:50.171155    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" path="/var/lib/kubelet/pods/0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea/volumes"
Nov 01 23:10:53.171843 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:10:53.171799    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:03.172000 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:03.171955    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:08.309083 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.309005    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:11:08.358332 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.358294    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\") pod \"2c305d46-d28a-4b6b-90eb-9663bd155d17\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") "
Nov 01 23:11:08.358519 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.358430    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\" (UniqueName: \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\") pod \"2c305d46-d28a-4b6b-90eb-9663bd155d17\" (UID: \"2c305d46-d28a-4b6b-90eb-9663bd155d17\") "
Nov 01 23:11:08.358755 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.358732    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume" pod "2c305d46-d28a-4b6b-90eb-9663bd155d17" (UID: "2c305d46-d28a-4b6b-90eb-9663bd155d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:08.358952 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.358931    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c305d46-d28a-4b6b-90eb-9663bd155d17-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:08.360710 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.360682    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452" pod "2c305d46-d28a-4b6b-90eb-9663bd155d17" (UID: "2c305d46-d28a-4b6b-90eb-9663bd155d17"). InnerVolumeSpecName "kube-api-access-dl452". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:08.396849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.396817    5031 generic.go:356] "Generic (PLEG): container finished" podID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerID="0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f" exitCode=0
Nov 01 23:11:08.397032 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.396863    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"
Nov 01 23:11:08.397032 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.396893    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" event={"ID":"2c305d46-d28a-4b6b-90eb-9663bd155d17","Type":"ContainerDied","Data":"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"}
Nov 01 23:11:08.397032 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.396946    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79" event={"ID":"2c305d46-d28a-4b6b-90eb-9663bd155d17","Type":"ContainerDied","Data":"622c6f40f1fd97b2a74628737d5f47cfb5526b12f8bc2ec4de512e9b32906513"}
Nov 01 23:11:08.397032 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.396967    5031 scope.go:122] "RemoveContainer" containerID="0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"
Nov 01 23:11:08.406550 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.406517    5031 scope.go:122] "RemoveContainer" containerID="0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"
Nov 01 23:11:08.406905 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:08.406885    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f\": container with ID starting with 0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f not found: ID does not exist" containerID="0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"
Nov 01 23:11:08.406963 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.406917    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f"} err="failed to get container status \"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f\": rpc error: code = NotFound desc = could not find container \"0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f\": container with ID starting with 0bbfd56350bcb425390b015e745899c1833b86978774cf3144a101033b54f68f not found: ID does not exist"
Nov 01 23:11:08.459924 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.459888    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl452\" (UniqueName: \"kubernetes.io/projected/2c305d46-d28a-4b6b-90eb-9663bd155d17-kube-api-access-dl452\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:08.518510 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.518474    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:11:08.522882 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:08.522850    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-675d1d-585f57dfcf-jfn79"]
Nov 01 23:11:09.271731 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:09.271692    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:11:09.272019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:09.271979    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" containerID="cri-o://f5301b85468f8cd212c89bf0a8e5be374190f7a7444b49821b1d4bb90cea90a0" gracePeriod=30
Nov 01 23:11:10.171071 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:10.171037    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" path="/var/lib/kubelet/pods/2c305d46-d28a-4b6b-90eb-9663bd155d17/volumes"
Nov 01 23:11:10.626773 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:10.626736    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:11:11.368721 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:11.368689    5031 patch_prober.go:29] interesting pod/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:11:11.369199 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:11.368745    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:11:14.436366 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.436335    5031 generic.go:356] "Generic (PLEG): container finished" podID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerID="f5301b85468f8cd212c89bf0a8e5be374190f7a7444b49821b1d4bb90cea90a0" exitCode=0
Nov 01 23:11:14.436852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.436407    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerDied","Data":"f5301b85468f8cd212c89bf0a8e5be374190f7a7444b49821b1d4bb90cea90a0"}
Nov 01 23:11:14.436852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.436474    5031 scope.go:122] "RemoveContainer" containerID="4817f55b047f2b88f68a9e6ce5b1efcf3a8a78422d159ac18eed64707abb9831"
Nov 01 23:11:14.552598 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.552568    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:11:14.626503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626465    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626519    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626581    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626622    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626650    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626708    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626743    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626781    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626851    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\") pod \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\" (UID: \"96e3c20d-d3a2-46d3-98c6-56ca0017a485\") "
Nov 01 23:11:14.626996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626832    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:14.626996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.626938    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:14.627261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627012    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:14.627261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627061    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:14.627261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627189    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:14.627448 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627430    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.627509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627450    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.627509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627459    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.627509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627471    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.627509 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.627484    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.629096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.629072    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:14.629237 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.629221    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:14.629618 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.629596    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:11:14.629691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.629639    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x" pod "96e3c20d-d3a2-46d3-98c6-56ca0017a485" (UID: "96e3c20d-d3a2-46d3-98c6-56ca0017a485"). InnerVolumeSpecName "kube-api-access-zfs4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:14.710212 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.710176    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 23:11:14.728674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.728633    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfs4x\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-kube-api-access-zfs4x\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.728674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.728665    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.728674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.728675    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.728674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.728683    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e3c20d-d3a2-46d3-98c6-56ca0017a485-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.757938 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:14.757899    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 23:11:15.443366 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:15.443325    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"
Nov 01 23:11:15.443862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:15.443369    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr" event={"ID":"96e3c20d-d3a2-46d3-98c6-56ca0017a485","Type":"ContainerDied","Data":"a534c597aeaf3bdf65e3fa99cc3fc947b133022df03f9555c03fae26b80dbde1"}
Nov 01 23:11:15.443862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:15.443435    5031 scope.go:122] "RemoveContainer" containerID="f5301b85468f8cd212c89bf0a8e5be374190f7a7444b49821b1d4bb90cea90a0"
Nov 01 23:11:15.598996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:15.598953    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:11:15.643893 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:15.643851    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3eb631-openshift-default-fc6fb9db7-kx4vr"]
Nov 01 23:11:16.173296 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:16.173253    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" path="/var/lib/kubelet/pods/96e3c20d-d3a2-46d3-98c6-56ca0017a485/volumes"
Nov 01 23:11:29.731452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:29.731367    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" containerID="cri-o://013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571" gracePeriod=45
Nov 01 23:11:29.780607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:29.780520    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" containerID="cri-o://273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e" gracePeriod=45
Nov 01 23:11:29.853622 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:29.853575    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": dial tcp 10.132.0.55:8000: connect: connection refused"
Nov 01 23:11:29.879458 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:29.879417    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": dial tcp 10.132.0.43:8000: connect: connection refused"
Nov 01 23:11:30.048241 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.048207    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 23:11:30.078502 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.078467    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 23:11:30.204668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204630    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.204668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204668    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.204929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204711    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.204929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204733    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\" (UniqueName: \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.204929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204805    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.204929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204836    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.204929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204913    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.204991    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205009    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205034    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205060    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205116    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\" (UniqueName: \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\") pod \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\" (UID: \"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b\") "
Nov 01 23:11:30.205201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205196    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\") pod \"8abdde21-b92d-44bf-8454-b346bf136a24\" (UID: \"8abdde21-b92d-44bf-8454-b346bf136a24\") "
Nov 01 23:11:30.205513 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205445    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.205632 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205602    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.205872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205777    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.205872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205796    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.205872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.205811    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.206080 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.206049    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.206404 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.206383    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.206508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.206448    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.209108 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209074    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.209108 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209082    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.209478 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209439    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "kube-api-access-krxjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:30.209638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209584    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "kube-api-access-x4lft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:30.209789 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209764    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs" pod "8abdde21-b92d-44bf-8454-b346bf136a24" (UID: "8abdde21-b92d-44bf-8454-b346bf136a24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:30.209789 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.209776    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs" pod "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" (UID: "b2aec9d0-70f3-4823-8fa8-bce08fc89c8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307342    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307376    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307393    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxjw\" (UniqueName: \"kubernetes.io/projected/8abdde21-b92d-44bf-8454-b346bf136a24-kube-api-access-krxjw\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307411    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307426    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307441    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdde21-b92d-44bf-8454-b346bf136a24-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307457    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdde21-b92d-44bf-8454-b346bf136a24-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307470    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.307780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.307485    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lft\" (UniqueName: \"kubernetes.io/projected/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b-kube-api-access-x4lft\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.531951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.531913    5031 generic.go:356] "Generic (PLEG): container finished" podID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerID="273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e" exitCode=0
Nov 01 23:11:30.531951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.531950    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"
Nov 01 23:11:30.532197 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.531992    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" event={"ID":"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b","Type":"ContainerDied","Data":"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"}
Nov 01 23:11:30.532197 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.532036    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898" event={"ID":"b2aec9d0-70f3-4823-8fa8-bce08fc89c8b","Type":"ContainerDied","Data":"b5ed07b690d10e7e0ae78f65ba47d3328b3d4e22e91ef49ad88675e47bc7295f"}
Nov 01 23:11:30.532197 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.532053    5031 scope.go:122] "RemoveContainer" containerID="273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"
Nov 01 23:11:30.533357 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.533336    5031 generic.go:356] "Generic (PLEG): container finished" podID="8abdde21-b92d-44bf-8454-b346bf136a24" containerID="013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571" exitCode=0
Nov 01 23:11:30.533439 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.533398    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"
Nov 01 23:11:30.533439 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.533418    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" event={"ID":"8abdde21-b92d-44bf-8454-b346bf136a24","Type":"ContainerDied","Data":"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"}
Nov 01 23:11:30.533575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.533453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt" event={"ID":"8abdde21-b92d-44bf-8454-b346bf136a24","Type":"ContainerDied","Data":"96d2d18896db1d6d651bfe0a039b2286f57d69d4b1fc601b685bcffffc89e580"}
Nov 01 23:11:30.543389 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.543368    5031 scope.go:122] "RemoveContainer" containerID="273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"
Nov 01 23:11:30.543747 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:30.543723    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e\": container with ID starting with 273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e not found: ID does not exist" containerID="273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"
Nov 01 23:11:30.543829 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.543756    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e"} err="failed to get container status \"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e\": rpc error: code = NotFound desc = could not find container \"273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e\": container with ID starting with 273ed6d5bc37a966e26587ecc06d572d4802d975e6ce639c3051850daedf344e not found: ID does not exist"
Nov 01 23:11:30.543829 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.543776    5031 scope.go:122] "RemoveContainer" containerID="013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"
Nov 01 23:11:30.552515 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.552484    5031 scope.go:122] "RemoveContainer" containerID="013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"
Nov 01 23:11:30.552886 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:30.552866    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571\": container with ID starting with 013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571 not found: ID does not exist" containerID="013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"
Nov 01 23:11:30.552938 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.552894    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571"} err="failed to get container status \"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571\": rpc error: code = NotFound desc = could not find container \"013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571\": container with ID starting with 013044e358df8b78cb739cd9f81754a2c76b470568d458178931768145b26571 not found: ID does not exist"
Nov 01 23:11:30.685843 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.685761    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 23:11:30.731686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.731644    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/simulator-w6-a6dd2f-kserve-6779c6b844-cv898"]
Nov 01 23:11:30.833715 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.833679    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 23:11:30.838506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:30.838467    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-2-w6-a6dd2f-kserve-548c685587-8wrnt"]
Nov 01 23:11:32.170479 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:32.170443    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" path="/var/lib/kubelet/pods/8abdde21-b92d-44bf-8454-b346bf136a24/volumes"
Nov 01 23:11:32.170933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:32.170908    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" path="/var/lib/kubelet/pods/b2aec9d0-70f3-4823-8fa8-bce08fc89c8b/volumes"
Nov 01 23:11:40.626987 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:40.626947    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:11:45.532574 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.532518    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:11:45.533005 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.532763    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" podUID="359a82d1-a799-4f64-ac60-2ddd85732fc3" containerName="maas-api" containerID="cri-o://79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4" gracePeriod=30
Nov 01 23:11:45.795969 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.795943    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:11:45.880877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.880840    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") pod \"359a82d1-a799-4f64-ac60-2ddd85732fc3\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") "
Nov 01 23:11:45.881047 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.880912    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\" (UniqueName: \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\") pod \"359a82d1-a799-4f64-ac60-2ddd85732fc3\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") "
Nov 01 23:11:45.881047 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.880964    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\") pod \"359a82d1-a799-4f64-ac60-2ddd85732fc3\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") "
Nov 01 23:11:45.881047 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.881015    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\") pod \"359a82d1-a799-4f64-ac60-2ddd85732fc3\" (UID: \"359a82d1-a799-4f64-ac60-2ddd85732fc3\") "
Nov 01 23:11:45.881444 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.881408    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca" pod "359a82d1-a799-4f64-ac60-2ddd85732fc3" (UID: "359a82d1-a799-4f64-ac60-2ddd85732fc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:45.883236 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.883208    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6" pod "359a82d1-a799-4f64-ac60-2ddd85732fc3" (UID: "359a82d1-a799-4f64-ac60-2ddd85732fc3"). InnerVolumeSpecName "kube-api-access-7d4m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:45.883794 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.883766    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs" pod "359a82d1-a799-4f64-ac60-2ddd85732fc3" (UID: "359a82d1-a799-4f64-ac60-2ddd85732fc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:45.883794 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.883777    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls" pod "359a82d1-a799-4f64-ac60-2ddd85732fc3" (UID: "359a82d1-a799-4f64-ac60-2ddd85732fc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:45.982141 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.982101    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:45.982141 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.982140    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359a82d1-a799-4f64-ac60-2ddd85732fc3-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:45.982141 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.982150    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359a82d1-a799-4f64-ac60-2ddd85732fc3-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:45.982397 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:45.982159    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d4m6\" (UniqueName: \"kubernetes.io/projected/359a82d1-a799-4f64-ac60-2ddd85732fc3-kube-api-access-7d4m6\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:46.624704 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.624667    5031 generic.go:356] "Generic (PLEG): container finished" podID="359a82d1-a799-4f64-ac60-2ddd85732fc3" containerID="79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4" exitCode=0
Nov 01 23:11:46.625153 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.624720    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"
Nov 01 23:11:46.625153 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.624758    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" event={"ID":"359a82d1-a799-4f64-ac60-2ddd85732fc3","Type":"ContainerDied","Data":"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"}
Nov 01 23:11:46.625153 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.624816    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf" event={"ID":"359a82d1-a799-4f64-ac60-2ddd85732fc3","Type":"ContainerDied","Data":"8c912bad77b55e1317205c934cb6fdb6b870af48ac5b95d5015963ae738070dc"}
Nov 01 23:11:46.625153 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.624839    5031 scope.go:122] "RemoveContainer" containerID="79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"
Nov 01 23:11:46.634181 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.634156    5031 scope.go:122] "RemoveContainer" containerID="79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"
Nov 01 23:11:46.634518 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:11:46.634498    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4\": container with ID starting with 79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4 not found: ID does not exist" containerID="79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"
Nov 01 23:11:46.634617 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.634530    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4"} err="failed to get container status \"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4\": rpc error: code = NotFound desc = could not find container \"79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4\": container with ID starting with 79bbad19c85a1b39a5d15c53ff819c092e7c25ba413f24a757b9faa035eee9b4 not found: ID does not exist"
Nov 01 23:11:46.777459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.777424    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:11:46.823565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:46.823503    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4af44d-7f7ffb5db-vs8hf"]
Nov 01 23:11:48.171418 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:11:48.171386    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359a82d1-a799-4f64-ac60-2ddd85732fc3" path="/var/lib/kubelet/pods/359a82d1-a799-4f64-ac60-2ddd85732fc3/volumes"
Nov 01 23:12:10.626081 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:10.626041    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:12:39.359806 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:39.359530    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 23:12:39.360628 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:39.359891    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" containerID="cri-o://6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041" gracePeriod=30
Nov 01 23:12:40.626891 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:40.626851    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:12:40.711189 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:40.711147    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-7pbvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:12:40.711368 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:40.711203    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:12:44.626377 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.626350    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 23:12:44.639760 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.639665    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 23:12:44.640048 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.639987    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" podUID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" containerName="maas-api" containerID="cri-o://400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208" gracePeriod=30
Nov 01 23:12:44.731869 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.731827    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732026 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.731881    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732026 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.731949    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732051    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732126    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732238 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732145    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:44.732238 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732238 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732232    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732262    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\") pod \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\" (UID: \"ea484b4f-95ce-499c-9c25-1ec54cc3967f\") "
Nov 01 23:12:44.732611 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732528    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:44.732611 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732582    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:44.732767 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732749    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:44.732930 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732913    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.733013 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732937    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.733013 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732961    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.733013 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.732975    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.733261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.733201    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:44.734978 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.734945    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:44.735262 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.735241    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:12:44.735358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.735330    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "kube-api-access-qk2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:44.735577 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.735528    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy" pod "ea484b4f-95ce-499c-9c25-1ec54cc3967f" (UID: "ea484b4f-95ce-499c-9c25-1ec54cc3967f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:44.834580 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.834502    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.834580 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.834560    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.834580 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.834577    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2nc\" (UniqueName: \"kubernetes.io/projected/ea484b4f-95ce-499c-9c25-1ec54cc3967f-kube-api-access-qk2nc\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.834933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.834592    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.834933 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.834604    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea484b4f-95ce-499c-9c25-1ec54cc3967f-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.907673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.907638    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 23:12:44.930694 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.930655    5031 generic.go:356] "Generic (PLEG): container finished" podID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" containerID="400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208" exitCode=0
Nov 01 23:12:44.930884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.930705    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"
Nov 01 23:12:44.930884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.930742    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" event={"ID":"f0e6e810-2bcf-4269-9106-7cf61f233eb7","Type":"ContainerDied","Data":"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"}
Nov 01 23:12:44.930884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.930801    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g" event={"ID":"f0e6e810-2bcf-4269-9106-7cf61f233eb7","Type":"ContainerDied","Data":"67f9c2566d373e2565ac0cc1d835d47492699d966bda326fd9ce80a0c187502d"}
Nov 01 23:12:44.930884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.930833    5031 scope.go:122] "RemoveContainer" containerID="400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"
Nov 01 23:12:44.932127 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.932101    5031 generic.go:356] "Generic (PLEG): container finished" podID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerID="6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041" exitCode=0
Nov 01 23:12:44.932243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.932132    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" event={"ID":"ea484b4f-95ce-499c-9c25-1ec54cc3967f","Type":"ContainerDied","Data":"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"}
Nov 01 23:12:44.932243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.932157    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"
Nov 01 23:12:44.932243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.932183    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq" event={"ID":"ea484b4f-95ce-499c-9c25-1ec54cc3967f","Type":"ContainerDied","Data":"f99907c800c3c420f9c7ba2e6c96ad08d5c43ae83ded1bef3e3ade2c904f35f9"}
Nov 01 23:12:44.935024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935004    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\") pod \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") "
Nov 01 23:12:44.935140 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935038    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\") pod \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") "
Nov 01 23:12:44.935140 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935100    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\" (UniqueName: \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\") pod \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") "
Nov 01 23:12:44.935266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935192    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") pod \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\" (UID: \"f0e6e810-2bcf-4269-9106-7cf61f233eb7\") "
Nov 01 23:12:44.935448 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935428    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca" pod "f0e6e810-2bcf-4269-9106-7cf61f233eb7" (UID: "f0e6e810-2bcf-4269-9106-7cf61f233eb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:44.935667 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.935632    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6e810-2bcf-4269-9106-7cf61f233eb7-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:44.937925 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.937900    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs" pod "f0e6e810-2bcf-4269-9106-7cf61f233eb7" (UID: "f0e6e810-2bcf-4269-9106-7cf61f233eb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:44.938012 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.937927    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls" pod "f0e6e810-2bcf-4269-9106-7cf61f233eb7" (UID: "f0e6e810-2bcf-4269-9106-7cf61f233eb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:44.942472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.942451    5031 scope.go:122] "RemoveContainer" containerID="400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"
Nov 01 23:12:44.942892 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:44.942869    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208\": container with ID starting with 400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208 not found: ID does not exist" containerID="400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"
Nov 01 23:12:44.942971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.942900    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208"} err="failed to get container status \"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208\": rpc error: code = NotFound desc = could not find container \"400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208\": container with ID starting with 400edf2ac796ee327a2eec31fb0c6b04807325b6296a693625d1f01749e28208 not found: ID does not exist"
Nov 01 23:12:44.942971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.942919    5031 scope.go:122] "RemoveContainer" containerID="6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"
Nov 01 23:12:44.943403 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.943375    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg" pod "f0e6e810-2bcf-4269-9106-7cf61f233eb7" (UID: "f0e6e810-2bcf-4269-9106-7cf61f233eb7"). InnerVolumeSpecName "kube-api-access-4qmqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:44.951868 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.951841    5031 scope.go:122] "RemoveContainer" containerID="6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"
Nov 01 23:12:44.952142 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:44.952117    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041\": container with ID starting with 6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041 not found: ID does not exist" containerID="6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"
Nov 01 23:12:44.952213 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:44.952167    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041"} err="failed to get container status \"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041\": rpc error: code = NotFound desc = could not find container \"6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041\": container with ID starting with 6743b7ad3c3aadb2210925fac78c48334b44bff017cd17bded57c89cf13a1041 not found: ID does not exist"
Nov 01 23:12:45.037122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.037086    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:45.037122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.037115    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e6e810-2bcf-4269-9106-7cf61f233eb7-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:45.037122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.037127    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qmqg\" (UniqueName: \"kubernetes.io/projected/f0e6e810-2bcf-4269-9106-7cf61f233eb7-kube-api-access-4qmqg\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:45.166049 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.165962    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 23:12:45.170037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.170007    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-7pbvq"]
Nov 01 23:12:45.344835 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.344799    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 23:12:45.349085 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:45.349058    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-646881-7494b8b7c-89h5g"]
Nov 01 23:12:46.171154 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:46.171107    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" path="/var/lib/kubelet/pods/ea484b4f-95ce-499c-9c25-1ec54cc3967f/volumes"
Nov 01 23:12:46.171761 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:46.171739    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" path="/var/lib/kubelet/pods/f0e6e810-2bcf-4269-9106-7cf61f233eb7/volumes"
Nov 01 23:12:52.378832 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:52.378799    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 23:12:52.379315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:52.379015    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy" containerID="cri-o://8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47" gracePeriod=30
Nov 01 23:12:57.646109 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.646042    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 23:12:57.767079 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767037    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767118    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767154    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767201    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767221    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767286 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767286    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767316    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767374    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767388    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:57.767603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767520    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:57.767603 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767582    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\") pod \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\" (UID: \"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5\") "
Nov 01 23:12:57.767895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767660    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:57.767895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767710    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:57.767895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.767841    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:57.768326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.768255    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.768326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.768283    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.768326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.768299    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.768326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.768313    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.768326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.768327    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.769739 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.769713    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:57.769861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.769716    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:57.769861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.769851    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "kube-api-access-99db8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:57.769940 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.769876    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo" pod "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" (UID: "fc855f75-bd1e-4b7b-8910-31c3cff3a1c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:12:57.869156 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.869116    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99db8\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-kube-api-access-99db8\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.869156 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.869152    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.869156 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.869161    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:57.869396 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:57.869170    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:58.004768 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.004673    5031 generic.go:356] "Generic (PLEG): container finished" podID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerID="8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47" exitCode=0
Nov 01 23:12:58.004768 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.004731    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"
Nov 01 23:12:58.005012 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.004768    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerDied","Data":"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"}
Nov 01 23:12:58.005012 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.004835    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg" event={"ID":"fc855f75-bd1e-4b7b-8910-31c3cff3a1c5","Type":"ContainerDied","Data":"06c9bd5d883d3f67e55728d6e9ac45b7966f2544d543da3f7f7c8158f77df99d"}
Nov 01 23:12:58.005012 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.004864    5031 scope.go:122] "RemoveContainer" containerID="8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"
Nov 01 23:12:58.014727 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.014706    5031 scope.go:122] "RemoveContainer" containerID="521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"
Nov 01 23:12:58.026730 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.026701    5031 scope.go:122] "RemoveContainer" containerID="8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"
Nov 01 23:12:58.027105 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:58.027081    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47\": container with ID starting with 8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47 not found: ID does not exist" containerID="8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"
Nov 01 23:12:58.027184 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.027120    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47"} err="failed to get container status \"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47\": rpc error: code = NotFound desc = could not find container \"8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47\": container with ID starting with 8b3d85f88c6469c795c85d2a1e889995026ae442440d483e3044f5efad866b47 not found: ID does not exist"
Nov 01 23:12:58.027184 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.027146    5031 scope.go:122] "RemoveContainer" containerID="521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"
Nov 01 23:12:58.027401 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:12:58.027381    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0\": container with ID starting with 521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0 not found: ID does not exist" containerID="521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"
Nov 01 23:12:58.027468 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.027411    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0"} err="failed to get container status \"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0\": rpc error: code = NotFound desc = could not find container \"521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0\": container with ID starting with 521de64c4e54b3885977591e858f0ab224063e89bf5f4dd4ab2ac51da36733b0 not found: ID does not exist"
Nov 01 23:12:58.158243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.158207    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 23:12:58.161974 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.161944    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rdrvg"]
Nov 01 23:12:58.170674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:12:58.170646    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" path="/var/lib/kubelet/pods/fc855f75-bd1e-4b7b-8910-31c3cff3a1c5/volumes"
Nov 01 23:13:06.335921 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:06.335886    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 23:13:10.627199 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:10.627161    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:13:10.794712 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:10.794675    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:13:10.794948 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:10.794921    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerName="maas-api" containerID="cri-o://82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526" gracePeriod=30
Nov 01 23:13:11.057939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.057911    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:13:11.072205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.072162    5031 generic.go:356] "Generic (PLEG): container finished" podID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerID="82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526" exitCode=0
Nov 01 23:13:11.072386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.072205    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" event={"ID":"a6aed3a7-7bfd-4c62-a636-df97ceb3211a","Type":"ContainerDied","Data":"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"}
Nov 01 23:13:11.072386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.072263    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" event={"ID":"a6aed3a7-7bfd-4c62-a636-df97ceb3211a","Type":"ContainerDied","Data":"0bf6dc3058db5a8a4aef994314bcdab3d15c55081ac3c1af6afadfbcff87b7b9"}
Nov 01 23:13:11.072386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.072217    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"
Nov 01 23:13:11.072386 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.072289    5031 scope.go:122] "RemoveContainer" containerID="82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"
Nov 01 23:13:11.081472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.081448    5031 scope.go:122] "RemoveContainer" containerID="82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"
Nov 01 23:13:11.081849 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:11.081827    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526\": container with ID starting with 82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526 not found: ID does not exist" containerID="82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"
Nov 01 23:13:11.081928 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.081860    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526"} err="failed to get container status \"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526\": rpc error: code = NotFound desc = could not find container \"82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526\": container with ID starting with 82327ec30b77e2578a52d55f1eb6e9493ab3717189c8646b0ef6b52cbf536526 not found: ID does not exist"
Nov 01 23:13:11.204851 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.204761    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\" (UniqueName: \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\") pod \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") "
Nov 01 23:13:11.204851 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.204814    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") pod \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") "
Nov 01 23:13:11.205096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.204939    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\") pod \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") "
Nov 01 23:13:11.205096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.205025    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\") pod \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\" (UID: \"a6aed3a7-7bfd-4c62-a636-df97ceb3211a\") "
Nov 01 23:13:11.205366 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.205334    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca" pod "a6aed3a7-7bfd-4c62-a636-df97ceb3211a" (UID: "a6aed3a7-7bfd-4c62-a636-df97ceb3211a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:11.207259 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.207232    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls" pod "a6aed3a7-7bfd-4c62-a636-df97ceb3211a" (UID: "a6aed3a7-7bfd-4c62-a636-df97ceb3211a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:11.207399 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.207372    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs" pod "a6aed3a7-7bfd-4c62-a636-df97ceb3211a" (UID: "a6aed3a7-7bfd-4c62-a636-df97ceb3211a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:11.207847 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.207828    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2" pod "a6aed3a7-7bfd-4c62-a636-df97ceb3211a" (UID: "a6aed3a7-7bfd-4c62-a636-df97ceb3211a"). InnerVolumeSpecName "kube-api-access-rq2c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:11.306993 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.306949    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.306993 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.306984    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.306993 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.307001    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq2c2\" (UniqueName: \"kubernetes.io/projected/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-kube-api-access-rq2c2\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.307236 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.307011    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6aed3a7-7bfd-4c62-a636-df97ceb3211a-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.336629 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.336585    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" podUID="52c8b445-4220-4796-8df4-bfc9b30bd951" containerName="payload-pre-processing" containerID="cri-o://a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e" gracePeriod=25
Nov 01 23:13:11.513941 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.513880    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:13:11.518902 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.518863    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh"]
Nov 01 23:13:11.574408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.574383    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 23:13:11.712335 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.712236    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\" (UniqueName: \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\") pod \"52c8b445-4220-4796-8df4-bfc9b30bd951\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") "
Nov 01 23:13:11.712792 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.712374    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\") pod \"52c8b445-4220-4796-8df4-bfc9b30bd951\" (UID: \"52c8b445-4220-4796-8df4-bfc9b30bd951\") "
Nov 01 23:13:11.712986 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.712946    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume" pod "52c8b445-4220-4796-8df4-bfc9b30bd951" (UID: "52c8b445-4220-4796-8df4-bfc9b30bd951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:11.714966 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.714931    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp" pod "52c8b445-4220-4796-8df4-bfc9b30bd951" (UID: "52c8b445-4220-4796-8df4-bfc9b30bd951"). InnerVolumeSpecName "kube-api-access-tw5dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:11.813567 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.813509    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52c8b445-4220-4796-8df4-bfc9b30bd951-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:11.813567 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:11.813567    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5dp\" (UniqueName: \"kubernetes.io/projected/52c8b445-4220-4796-8df4-bfc9b30bd951-kube-api-access-tw5dp\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:12.079416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.079378    5031 generic.go:356] "Generic (PLEG): container finished" podID="52c8b445-4220-4796-8df4-bfc9b30bd951" containerID="a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e" exitCode=0
Nov 01 23:13:12.079661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.079425    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"
Nov 01 23:13:12.079661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.079453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" event={"ID":"52c8b445-4220-4796-8df4-bfc9b30bd951","Type":"ContainerDied","Data":"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"}
Nov 01 23:13:12.079661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.079506    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5" event={"ID":"52c8b445-4220-4796-8df4-bfc9b30bd951","Type":"ContainerDied","Data":"638a1614b738a2950b5eec1a7c597a68e5c5cac4f8f1b53d4e7341246ed2feeb"}
Nov 01 23:13:12.079661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.079530    5031 scope.go:122] "RemoveContainer" containerID="a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"
Nov 01 23:13:12.089610 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.089583    5031 scope.go:122] "RemoveContainer" containerID="a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"
Nov 01 23:13:12.089954 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:12.089932    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e\": container with ID starting with a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e not found: ID does not exist" containerID="a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"
Nov 01 23:13:12.090008 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.089966    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e"} err="failed to get container status \"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e\": rpc error: code = NotFound desc = could not find container \"a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e\": container with ID starting with a37827ea19a86aa7d99c87d55b1d9a4b549a71f673b539f01c6f3bf965cb170e not found: ID does not exist"
Nov 01 23:13:12.170211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.170173    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" path="/var/lib/kubelet/pods/a6aed3a7-7bfd-4c62-a636-df97ceb3211a/volumes"
Nov 01 23:13:12.232838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.232806    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 23:13:12.235902 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:12.235871    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8cc47a-f855d4b99-ds8b5"]
Nov 01 23:13:14.049209 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:14.049153    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf46c7-8dfd7d866-lclgh" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.108:8443/health\": context deadline exceeded"
Nov 01 23:13:14.169983 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:14.169936    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c8b445-4220-4796-8df4-bfc9b30bd951" path="/var/lib/kubelet/pods/52c8b445-4220-4796-8df4-bfc9b30bd951/volumes"
Nov 01 23:13:32.398171 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:32.398133    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 23:13:32.503067 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:32.503019    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 23:13:37.399431 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.399362    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" podUID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" containerName="payload-processing" containerID="cri-o://50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62" gracePeriod=25
Nov 01 23:13:37.503986 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.503937    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" podUID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" containerName="payload-pre-processing" containerID="cri-o://b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f" gracePeriod=25
Nov 01 23:13:37.667104 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.667024    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 23:13:37.764435 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.764409    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 23:13:37.783291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.783257    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\") pod \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") "
Nov 01 23:13:37.783458 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.783325    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\" (UniqueName: \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\") pod \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\" (UID: \"d45d2f1a-36c0-4f86-bd72-50010f398ac7\") "
Nov 01 23:13:37.783788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.783757    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume" pod "d45d2f1a-36c0-4f86-bd72-50010f398ac7" (UID: "d45d2f1a-36c0-4f86-bd72-50010f398ac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:37.787220 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.787184    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v" pod "d45d2f1a-36c0-4f86-bd72-50010f398ac7" (UID: "d45d2f1a-36c0-4f86-bd72-50010f398ac7"). InnerVolumeSpecName "kube-api-access-ck84v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:37.885221 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885186    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\") pod \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") "
Nov 01 23:13:37.885404 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885337    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\" (UniqueName: \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\") pod \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\" (UID: \"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8\") "
Nov 01 23:13:37.885656 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885628    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume" pod "7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" (UID: "7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:37.885794 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885769    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45d2f1a-36c0-4f86-bd72-50010f398ac7-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:37.885864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885793    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck84v\" (UniqueName: \"kubernetes.io/projected/d45d2f1a-36c0-4f86-bd72-50010f398ac7-kube-api-access-ck84v\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:37.885864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.885807    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:37.888050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.888018    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64" pod "7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" (UID: "7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8"). InnerVolumeSpecName "kube-api-access-qvm64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:37.986979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:37.986871    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvm64\" (UniqueName: \"kubernetes.io/projected/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8-kube-api-access-qvm64\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:38.217844 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.217809    5031 generic.go:356] "Generic (PLEG): container finished" podID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" containerID="50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62" exitCode=0
Nov 01 23:13:38.218054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.217868    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"
Nov 01 23:13:38.218054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.217897    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" event={"ID":"d45d2f1a-36c0-4f86-bd72-50010f398ac7","Type":"ContainerDied","Data":"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"}
Nov 01 23:13:38.218054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.217947    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6" event={"ID":"d45d2f1a-36c0-4f86-bd72-50010f398ac7","Type":"ContainerDied","Data":"9eff532b9ebd409467b7dc3d9dfee1cf489a524c7ee40d1a2330c5bd94e443c9"}
Nov 01 23:13:38.218054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.217968    5031 scope.go:122] "RemoveContainer" containerID="50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"
Nov 01 23:13:38.219388 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.219363    5031 generic.go:356] "Generic (PLEG): container finished" podID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" containerID="b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f" exitCode=0
Nov 01 23:13:38.219499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.219421    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"
Nov 01 23:13:38.219499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.219453    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" event={"ID":"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8","Type":"ContainerDied","Data":"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"}
Nov 01 23:13:38.219499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.219491    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5" event={"ID":"7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8","Type":"ContainerDied","Data":"cca74b53251bccd0b4cfb3eb8ff359385873ec2b72aecaf1316c0961099fd295"}
Nov 01 23:13:38.227205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.227181    5031 scope.go:122] "RemoveContainer" containerID="50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"
Nov 01 23:13:38.227523 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:38.227493    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62\": container with ID starting with 50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62 not found: ID does not exist" containerID="50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"
Nov 01 23:13:38.227607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.227532    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62"} err="failed to get container status \"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62\": rpc error: code = NotFound desc = could not find container \"50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62\": container with ID starting with 50f56ea0d3ecdea59f656700e1428228f834b615318d74cc7f5e703b95d20b62 not found: ID does not exist"
Nov 01 23:13:38.227607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.227568    5031 scope.go:122] "RemoveContainer" containerID="b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"
Nov 01 23:13:38.235906 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.235885    5031 scope.go:122] "RemoveContainer" containerID="b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"
Nov 01 23:13:38.236216 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:38.236197    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f\": container with ID starting with b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f not found: ID does not exist" containerID="b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"
Nov 01 23:13:38.236273 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.236224    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f"} err="failed to get container status \"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f\": rpc error: code = NotFound desc = could not find container \"b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f\": container with ID starting with b48b0dc7902e9e30b3c01e41dbe919f6fab62e26dc3978ac0e992ed93473e96f not found: ID does not exist"
Nov 01 23:13:38.369328 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.369296    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 23:13:38.415291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.415260    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-8bac2f-6547fd6f45-8lbn6"]
Nov 01 23:13:38.474289 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.474257    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 23:13:38.478329 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:38.478297    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8bac2f-7cb9789f5-b6rj5"]
Nov 01 23:13:40.170665 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:40.170629    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" path="/var/lib/kubelet/pods/7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8/volumes"
Nov 01 23:13:40.171067 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:40.171015    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" path="/var/lib/kubelet/pods/d45d2f1a-36c0-4f86-bd72-50010f398ac7/volumes"
Nov 01 23:13:40.625999 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:40.625953    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:13:43.969774 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:43.969736    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 23:13:43.970344 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:43.970006    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" podUID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" containerName="maas-api" containerID="cri-o://0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea" gracePeriod=30
Nov 01 23:13:44.222116 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.222042    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 23:13:44.252370 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.252336    5031 generic.go:356] "Generic (PLEG): container finished" podID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" containerID="0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea" exitCode=0
Nov 01 23:13:44.252586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.252380    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"
Nov 01 23:13:44.252586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.252426    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" event={"ID":"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6","Type":"ContainerDied","Data":"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"}
Nov 01 23:13:44.252586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.252477    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74" event={"ID":"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6","Type":"ContainerDied","Data":"d619977251bd6362b044696c1a953209fc8f472248ee6da589dd7a334d91a054"}
Nov 01 23:13:44.252586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.252500    5031 scope.go:122] "RemoveContainer" containerID="0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"
Nov 01 23:13:44.261464 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.261445    5031 scope.go:122] "RemoveContainer" containerID="0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"
Nov 01 23:13:44.261766 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:44.261748    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea\": container with ID starting with 0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea not found: ID does not exist" containerID="0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"
Nov 01 23:13:44.261848 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.261776    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea"} err="failed to get container status \"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea\": rpc error: code = NotFound desc = could not find container \"0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea\": container with ID starting with 0f647e37007251c3e16c8f21bb690560b6027f671023e2e065b00d5e14f547ea not found: ID does not exist"
Nov 01 23:13:44.346503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.346469    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\" (UniqueName: \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\") pod \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") "
Nov 01 23:13:44.346723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.346520    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\") pod \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") "
Nov 01 23:13:44.346723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.346706    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") pod \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") "
Nov 01 23:13:44.346806 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.346762    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\") pod \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\" (UID: \"d4bcb313-b6e0-4fe2-a82e-900e3f715dc6\") "
Nov 01 23:13:44.347320 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.347287    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca" pod "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" (UID: "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:44.349212 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.349180    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls" pod "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" (UID: "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:44.349481 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.349464    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl" pod "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" (UID: "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6"). InnerVolumeSpecName "kube-api-access-lcfjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:44.349600 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.349585    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs" pod "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" (UID: "d4bcb313-b6e0-4fe2-a82e-900e3f715dc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:44.448060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.448016    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:44.448060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.448055    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:44.448060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.448065    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfjl\" (UniqueName: \"kubernetes.io/projected/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-kube-api-access-lcfjl\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:44.448315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.448075    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:44.668059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.668024    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 23:13:44.670050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:44.670021    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8bac2f-b4b8467f4-5hz74"]
Nov 01 23:13:46.177090 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:46.177011    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" path="/var/lib/kubelet/pods/d4bcb313-b6e0-4fe2-a82e-900e3f715dc6/volumes"
Nov 01 23:13:52.595845 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:52.595775    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 23:13:52.596361 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:52.596081    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy" containerID="cri-o://93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0" gracePeriod=30
Nov 01 23:13:57.858058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.858023    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 23:13:57.884010 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.883923    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884010 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.883976    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884022    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884061    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884086    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884131    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884162    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884228 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884213    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:57.884596 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884244    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884596 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884295    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\") pod \"9b276892-0640-470f-ba01-70d5f06015ad\" (UID: \"9b276892-0640-470f-ba01-70d5f06015ad\") "
Nov 01 23:13:57.884596 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884468    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:57.884596 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884486    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:57.884823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884716    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:57.884823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.884732    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:57.885045 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.885025    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.885085 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.885053    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.885085 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.885070    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.885148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.885083    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b276892-0640-470f-ba01-70d5f06015ad-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.885148 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.885097    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.886519 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.886491    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:57.886662 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.886530    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:13:57.886893 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.886879    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:57.887005 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.886991    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z" pod "9b276892-0640-470f-ba01-70d5f06015ad" (UID: "9b276892-0640-470f-ba01-70d5f06015ad"). InnerVolumeSpecName "kube-api-access-8q42z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:57.986094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.986057    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q42z\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-kube-api-access-8q42z\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.986094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.986090    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b276892-0640-470f-ba01-70d5f06015ad-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.986094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.986099    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b276892-0640-470f-ba01-70d5f06015ad-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:57.986350 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:57.986108    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b276892-0640-470f-ba01-70d5f06015ad-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:58.323661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.323605    5031 generic.go:356] "Generic (PLEG): container finished" podID="9b276892-0640-470f-ba01-70d5f06015ad" containerID="93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0" exitCode=0
Nov 01 23:13:58.323838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.323671    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"
Nov 01 23:13:58.323838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.323699    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" event={"ID":"9b276892-0640-470f-ba01-70d5f06015ad","Type":"ContainerDied","Data":"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"}
Nov 01 23:13:58.323838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.323759    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q" event={"ID":"9b276892-0640-470f-ba01-70d5f06015ad","Type":"ContainerDied","Data":"0ce108c9ae716713b774dfeb77e094799bebe169c9b7356c4b68051ad8ec6276"}
Nov 01 23:13:58.323838 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.323782    5031 scope.go:122] "RemoveContainer" containerID="93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"
Nov 01 23:13:58.333207 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.333178    5031 scope.go:122] "RemoveContainer" containerID="93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"
Nov 01 23:13:58.333562 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:13:58.333519    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0\": container with ID starting with 93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0 not found: ID does not exist" containerID="93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"
Nov 01 23:13:58.333633 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.333572    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0"} err="failed to get container status \"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0\": rpc error: code = NotFound desc = could not find container \"93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0\": container with ID starting with 93730b271684250e660342cbeb137612d5b1c40b02aa7e162f2be79da8b32cc0 not found: ID does not exist"
Nov 01 23:13:58.476855 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.476770    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 23:13:58.480794 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:13:58.480770    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-j2h2q"]
Nov 01 23:14:00.170649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:00.170611    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b276892-0640-470f-ba01-70d5f06015ad" path="/var/lib/kubelet/pods/9b276892-0640-470f-ba01-70d5f06015ad/volumes"
Nov 01 23:14:05.001582 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:05.001527    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 23:14:05.909876 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:05.909836    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 23:14:05.910099 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:05.910077    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" podUID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" containerName="maas-api" containerID="cri-o://879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2" gracePeriod=30
Nov 01 23:14:06.172360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.172296    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 23:14:06.273446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.273407    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\" (UniqueName: \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\") pod \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") "
Nov 01 23:14:06.273446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.273448    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\") pod \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") "
Nov 01 23:14:06.273739 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.273477    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\") pod \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") "
Nov 01 23:14:06.273739 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.273584    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") pod \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\" (UID: \"cc6c6195-bd41-45b2-8ece-379c55f3f0a7\") "
Nov 01 23:14:06.273877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.273851    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca" pod "cc6c6195-bd41-45b2-8ece-379c55f3f0a7" (UID: "cc6c6195-bd41-45b2-8ece-379c55f3f0a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:06.275802 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.275776    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96" pod "cc6c6195-bd41-45b2-8ece-379c55f3f0a7" (UID: "cc6c6195-bd41-45b2-8ece-379c55f3f0a7"). InnerVolumeSpecName "kube-api-access-8xp96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:06.275890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.275817    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls" pod "cc6c6195-bd41-45b2-8ece-379c55f3f0a7" (UID: "cc6c6195-bd41-45b2-8ece-379c55f3f0a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:06.275890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.275855    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs" pod "cc6c6195-bd41-45b2-8ece-379c55f3f0a7" (UID: "cc6c6195-bd41-45b2-8ece-379c55f3f0a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:06.364340 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.364304    5031 generic.go:356] "Generic (PLEG): container finished" podID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" containerID="879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2" exitCode=0
Nov 01 23:14:06.364560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.364352    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"
Nov 01 23:14:06.364560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.364391    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" event={"ID":"cc6c6195-bd41-45b2-8ece-379c55f3f0a7","Type":"ContainerDied","Data":"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"}
Nov 01 23:14:06.364560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.364463    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn" event={"ID":"cc6c6195-bd41-45b2-8ece-379c55f3f0a7","Type":"ContainerDied","Data":"abfb355a32d27a9ffa5cd4dd419592864d2f759d5f50c03129c327e440ebd095"}
Nov 01 23:14:06.364560 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.364493    5031 scope.go:122] "RemoveContainer" containerID="879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"
Nov 01 23:14:06.374643 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.374619    5031 scope.go:122] "RemoveContainer" containerID="879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"
Nov 01 23:14:06.374968 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:06.374946    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2\": container with ID starting with 879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2 not found: ID does not exist" containerID="879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"
Nov 01 23:14:06.375034 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.374980    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2"} err="failed to get container status \"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2\": rpc error: code = NotFound desc = could not find container \"879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2\": container with ID starting with 879aac7008431a05146177d42751b565443cd55eefd663056d90021c1c389ca2 not found: ID does not exist"
Nov 01 23:14:06.375034 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.375027    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xp96\" (UniqueName: \"kubernetes.io/projected/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-kube-api-access-8xp96\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:06.375111 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.375042    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:06.375111 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.375053    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:06.375111 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.375063    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc6c6195-bd41-45b2-8ece-379c55f3f0a7-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:06.515554 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.515459    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 23:14:06.516893 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:06.516863    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-661e9f-7c5c8fd74d-wj6fn"]
Nov 01 23:14:07.043977 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.043937    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 23:14:07.044178 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.044152    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" podUID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" containerName="maas-api" containerID="cri-o://6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa" gracePeriod=30
Nov 01 23:14:07.303693 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.303619    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 23:14:07.370772 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.370736    5031 generic.go:356] "Generic (PLEG): container finished" podID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" containerID="6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa" exitCode=0
Nov 01 23:14:07.370772 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.370780    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"
Nov 01 23:14:07.371055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.370823    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" event={"ID":"0a8d188a-d423-46b7-8370-efa2eb45ba5e","Type":"ContainerDied","Data":"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"}
Nov 01 23:14:07.371055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.370882    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f" event={"ID":"0a8d188a-d423-46b7-8370-efa2eb45ba5e","Type":"ContainerDied","Data":"9376ad47e8b0bb39b76c522a6fbda43af39df8e17e597bafa63064f0b40005cc"}
Nov 01 23:14:07.371055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.370909    5031 scope.go:122] "RemoveContainer" containerID="6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"
Nov 01 23:14:07.380899 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.380878    5031 scope.go:122] "RemoveContainer" containerID="6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"
Nov 01 23:14:07.381212 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:07.381189    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa\": container with ID starting with 6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa not found: ID does not exist" containerID="6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"
Nov 01 23:14:07.381272 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.381221    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa"} err="failed to get container status \"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa\": rpc error: code = NotFound desc = could not find container \"6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa\": container with ID starting with 6bc5a1243b48a0553a96c9ac6c27ed95b9b6553ff4ebb16183efa8e707aa26aa not found: ID does not exist"
Nov 01 23:14:07.382490 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.382469    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\") pod \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") "
Nov 01 23:14:07.382616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.382532    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") pod \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") "
Nov 01 23:14:07.382685 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.382625    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\" (UniqueName: \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\") pod \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") "
Nov 01 23:14:07.382741 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.382690    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\") pod \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\" (UID: \"0a8d188a-d423-46b7-8370-efa2eb45ba5e\") "
Nov 01 23:14:07.382890 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.382867    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca" pod "0a8d188a-d423-46b7-8370-efa2eb45ba5e" (UID: "0a8d188a-d423-46b7-8370-efa2eb45ba5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:07.383022 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.383006    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8d188a-d423-46b7-8370-efa2eb45ba5e-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:07.384909 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.384884    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls" pod "0a8d188a-d423-46b7-8370-efa2eb45ba5e" (UID: "0a8d188a-d423-46b7-8370-efa2eb45ba5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:07.385010 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.384972    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs" pod "0a8d188a-d423-46b7-8370-efa2eb45ba5e" (UID: "0a8d188a-d423-46b7-8370-efa2eb45ba5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:07.385010 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.384988    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh" pod "0a8d188a-d423-46b7-8370-efa2eb45ba5e" (UID: "0a8d188a-d423-46b7-8370-efa2eb45ba5e"). InnerVolumeSpecName "kube-api-access-ls6sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:07.483516 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.483477    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:07.483516 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.483509    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8d188a-d423-46b7-8370-efa2eb45ba5e-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:07.483516 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.483518    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls6sh\" (UniqueName: \"kubernetes.io/projected/0a8d188a-d423-46b7-8370-efa2eb45ba5e-kube-api-access-ls6sh\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:07.782218 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.782178    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 23:14:07.828266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:07.828224    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c768f3-d7d4cd4d4-r4n9f"]
Nov 01 23:14:08.170066 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:08.170032    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" path="/var/lib/kubelet/pods/0a8d188a-d423-46b7-8370-efa2eb45ba5e/volumes"
Nov 01 23:14:08.170451 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:08.170436    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" path="/var/lib/kubelet/pods/cc6c6195-bd41-45b2-8ece-379c55f3f0a7/volumes"
Nov 01 23:14:10.002605 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.002531    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" podUID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" containerName="payload-pre-processing" containerID="cri-o://ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7" gracePeriod=25
Nov 01 23:14:10.258259 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.258184    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 23:14:10.309267 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.309234    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\" (UniqueName: \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\") pod \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") "
Nov 01 23:14:10.309441 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.309333    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\") pod \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\" (UID: \"95c5c865-ba7b-484d-9550-d1d181ff6ef4\") "
Nov 01 23:14:10.309856 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.309825    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume" pod "95c5c865-ba7b-484d-9550-d1d181ff6ef4" (UID: "95c5c865-ba7b-484d-9550-d1d181ff6ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:10.311736 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.311706    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7" pod "95c5c865-ba7b-484d-9550-d1d181ff6ef4" (UID: "95c5c865-ba7b-484d-9550-d1d181ff6ef4"). InnerVolumeSpecName "kube-api-access-dbjj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:10.398323 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.398289    5031 generic.go:356] "Generic (PLEG): container finished" podID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" containerID="ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7" exitCode=0
Nov 01 23:14:10.398480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.398340    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"
Nov 01 23:14:10.398480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.398368    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" event={"ID":"95c5c865-ba7b-484d-9550-d1d181ff6ef4","Type":"ContainerDied","Data":"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"}
Nov 01 23:14:10.398480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.398429    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx" event={"ID":"95c5c865-ba7b-484d-9550-d1d181ff6ef4","Type":"ContainerDied","Data":"1b65d64cbbcd336c35aec562d2adb2f39a9bd0c69d6a5417b00a5f6c9d530b1f"}
Nov 01 23:14:10.398480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.398461    5031 scope.go:122] "RemoveContainer" containerID="ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"
Nov 01 23:14:10.409586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.409558    5031 scope.go:122] "RemoveContainer" containerID="ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"
Nov 01 23:14:10.409961 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:10.409941    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7\": container with ID starting with ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7 not found: ID does not exist" containerID="ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"
Nov 01 23:14:10.410036 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.409977    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7"} err="failed to get container status \"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7\": rpc error: code = NotFound desc = could not find container \"ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7\": container with ID starting with ce10911b12bc3b7b3c68ee74546f618ad06c7dec6593a0c98b739b5fff7425f7 not found: ID does not exist"
Nov 01 23:14:10.410214 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.410195    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjj7\" (UniqueName: \"kubernetes.io/projected/95c5c865-ba7b-484d-9550-d1d181ff6ef4-kube-api-access-dbjj7\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:10.410259 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.410219    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5c865-ba7b-484d-9550-d1d181ff6ef4-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:10.550741 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.550703    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 23:14:10.555791 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:10.555756    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-661e9f-67d5bb64bc-q98fx"]
Nov 01 23:14:10.625137 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:10.625099    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:14:12.169917 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:12.169879    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" path="/var/lib/kubelet/pods/95c5c865-ba7b-484d-9550-d1d181ff6ef4/volumes"
Nov 01 23:14:22.646356 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:22.646319    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 23:14:22.692648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:22.692606    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 23:14:37.666954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:37.666883    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" containerID="cri-o://6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729" gracePeriod=45
Nov 01 23:14:37.713088 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:37.713045    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main" containerID="cri-o://07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4" gracePeriod=45
Nov 01 23:14:37.805928 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:37.805888    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": dial tcp 10.132.0.51:8000: connect: connection refused"
Nov 01 23:14:37.959748 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:37.959719    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 23:14:37.994819 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:37.994786    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 23:14:38.084483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084428    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.084667 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084559    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.084667 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084594    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.084667 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084647    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.084849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084702    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.084849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084721    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.084849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.084742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\" (UniqueName: \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\") pod \"7743905f-5888-4188-8ace-fd72d94cdec3\" (UID: \"7743905f-5888-4188-8ace-fd72d94cdec3\") "
Nov 01 23:14:38.085234 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.085205    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.086229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.085925    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.086229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.086162    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.087305 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.087282    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.087305 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.087298    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:38.087885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.087867    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm" pod "7743905f-5888-4188-8ace-fd72d94cdec3" (UID: "7743905f-5888-4188-8ace-fd72d94cdec3"). InnerVolumeSpecName "kube-api-access-w8ncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:38.185579 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185526    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.185750 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185637    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.185836 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185752    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.185836 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185799    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.185962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185828    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.185962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.185940    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\" (UniqueName: \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\") pod \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\" (UID: \"7d4580fb-35f4-4d7f-afcc-a00232349ce8\") "
Nov 01 23:14:38.186282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186246    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.186282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186269    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ncm\" (UniqueName: \"kubernetes.io/projected/7743905f-5888-4188-8ace-fd72d94cdec3-kube-api-access-w8ncm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.186282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186279    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.186450 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186289    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7743905f-5888-4188-8ace-fd72d94cdec3-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.186450 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186296    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7743905f-5888-4188-8ace-fd72d94cdec3-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.186530 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.186483    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.189581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.187780    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.189581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.188796    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.189581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.189174    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.190161 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.190129    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:38.190461 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.190434    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr" pod "7d4580fb-35f4-4d7f-afcc-a00232349ce8" (UID: "7d4580fb-35f4-4d7f-afcc-a00232349ce8"). InnerVolumeSpecName "kube-api-access-hm7sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:38.287301 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287263    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7sr\" (UniqueName: \"kubernetes.io/projected/7d4580fb-35f4-4d7f-afcc-a00232349ce8-kube-api-access-hm7sr\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.287301 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287296    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.287301 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287311    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.287593 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287322    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.287593 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287335    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d4580fb-35f4-4d7f-afcc-a00232349ce8-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.287593 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.287346    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d4580fb-35f4-4d7f-afcc-a00232349ce8-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.533214 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.533124    5031 generic.go:356] "Generic (PLEG): container finished" podID="7743905f-5888-4188-8ace-fd72d94cdec3" containerID="6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729" exitCode=0
Nov 01 23:14:38.533214 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.533184    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"
Nov 01 23:14:38.533472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.533210    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" event={"ID":"7743905f-5888-4188-8ace-fd72d94cdec3","Type":"ContainerDied","Data":"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"}
Nov 01 23:14:38.533472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.533255    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf" event={"ID":"7743905f-5888-4188-8ace-fd72d94cdec3","Type":"ContainerDied","Data":"8a93743d9d15428f3f19abc7d3be0ce3edd90cdf6f6105e75010a2cf5a2c3acd"}
Nov 01 23:14:38.533472 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.533271    5031 scope.go:122] "RemoveContainer" containerID="6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"
Nov 01 23:14:38.534786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.534759    5031 generic.go:356] "Generic (PLEG): container finished" podID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerID="07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4" exitCode=0
Nov 01 23:14:38.534906 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.534805    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"
Nov 01 23:14:38.534906 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.534831    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" event={"ID":"7d4580fb-35f4-4d7f-afcc-a00232349ce8","Type":"ContainerDied","Data":"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"}
Nov 01 23:14:38.534906 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.534859    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm" event={"ID":"7d4580fb-35f4-4d7f-afcc-a00232349ce8","Type":"ContainerDied","Data":"361d06de2f5ac55ce55e2ce0095e67ae9610909b365625ed3d5b9e63f68cd2f5"}
Nov 01 23:14:38.542974 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.542953    5031 scope.go:122] "RemoveContainer" containerID="6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"
Nov 01 23:14:38.543351 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:38.543328    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729\": container with ID starting with 6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729 not found: ID does not exist" containerID="6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"
Nov 01 23:14:38.543459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.543360    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729"} err="failed to get container status \"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729\": rpc error: code = NotFound desc = could not find container \"6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729\": container with ID starting with 6f1e94821ff0507a697bdd3145e843a7c9fa810c8b679f0497105203a1d8e729 not found: ID does not exist"
Nov 01 23:14:38.543459 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.543380    5031 scope.go:122] "RemoveContainer" containerID="07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"
Nov 01 23:14:38.551570 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.551517    5031 scope.go:122] "RemoveContainer" containerID="07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"
Nov 01 23:14:38.552271 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:38.552249    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4\": container with ID starting with 07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4 not found: ID does not exist" containerID="07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"
Nov 01 23:14:38.552358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.552279    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4"} err="failed to get container status \"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4\": rpc error: code = NotFound desc = could not find container \"07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4\": container with ID starting with 07561f60ca6a9e1a293002294d2d27b98f88a701ed3c27a0457e2b6fcf95c6d4 not found: ID does not exist"
Nov 01 23:14:38.685513 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.685482    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 23:14:38.731497 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.731464    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/premium-w4-9dfe9f-kserve-f5fcffb95-n99gm"]
Nov 01 23:14:38.792969 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.792941    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 23:14:38.798724 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:38.798694    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-w4-9dfe9f-kserve-7889c58778-s6dpf"]
Nov 01 23:14:40.169593 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:40.169561    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" path="/var/lib/kubelet/pods/7743905f-5888-4188-8ace-fd72d94cdec3/volumes"
Nov 01 23:14:40.169985 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:40.169971    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" path="/var/lib/kubelet/pods/7d4580fb-35f4-4d7f-afcc-a00232349ce8/volumes"
Nov 01 23:14:40.626077 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:14:40.626034    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:14:53.387282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.387235    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 23:14:53.387897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.387624    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" podUID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" containerName="maas-api" containerID="cri-o://c0a6655a556f6afea9a183ac7c7ef47cd3c659ad02fb69be51c8b62fe2b17542" gracePeriod=30
Nov 01 23:14:53.608084 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.608051    5031 generic.go:356] "Generic (PLEG): container finished" podID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" containerID="c0a6655a556f6afea9a183ac7c7ef47cd3c659ad02fb69be51c8b62fe2b17542" exitCode=0
Nov 01 23:14:53.608318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.608152    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" event={"ID":"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b","Type":"ContainerDied","Data":"c0a6655a556f6afea9a183ac7c7ef47cd3c659ad02fb69be51c8b62fe2b17542"}
Nov 01 23:14:53.653927 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.653902    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 23:14:53.712279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.712242    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\") pod \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") "
Nov 01 23:14:53.712498 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.712410    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\" (UniqueName: \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\") pod \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") "
Nov 01 23:14:53.712498 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.712483    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\") pod \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") "
Nov 01 23:14:53.712672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.712518    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") pod \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\" (UID: \"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b\") "
Nov 01 23:14:53.712894 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.712870    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca" pod "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" (UID: "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:53.714648 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.714625    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv" pod "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" (UID: "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b"). InnerVolumeSpecName "kube-api-access-7znnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:53.714722 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.714695    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls" pod "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" (UID: "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:53.714990 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.714968    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs" pod "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" (UID: "03d934bf-46d8-4ec1-92d9-3ba4c7fb032b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:53.813668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.813630    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7znnv\" (UniqueName: \"kubernetes.io/projected/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-kube-api-access-7znnv\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:53.813668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.813660    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:53.813668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.813670    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:53.813668 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:53.813679    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:54.613670 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:54.613630    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"
Nov 01 23:14:54.614150 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:54.613667    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv" event={"ID":"03d934bf-46d8-4ec1-92d9-3ba4c7fb032b","Type":"ContainerDied","Data":"35702f0bd9d84433f77072e6b1d2328f25935302318e0b5d1f0d8c7c3c51bb90"}
Nov 01 23:14:54.614150 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:54.613733    5031 scope.go:122] "RemoveContainer" containerID="c0a6655a556f6afea9a183ac7c7ef47cd3c659ad02fb69be51c8b62fe2b17542"
Nov 01 23:14:54.764576 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:54.764520    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 23:14:54.810718 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:54.810681    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-9dfe9f-6fdc446f94-nh2kv"]
Nov 01 23:14:56.169942 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:14:56.169908    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" path="/var/lib/kubelet/pods/03d934bf-46d8-4ec1-92d9-3ba4c7fb032b/volumes"
Nov 01 23:15:10.627345 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:15:10.627305    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:15:40.625796 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:15:40.625708    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:16:10.625653 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:16:10.625614    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:16:40.625475 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:16:40.625435    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:17:10.524420 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524383    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"]
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524746    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524757    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524765    5031 cpu_manager.go:414] "containerMap: removing container" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524771    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524778    5031 cpu_manager.go:414] "containerMap: removing container" podUID="359a82d1-a799-4f64-ac60-2ddd85732fc3" containerName="maas-api"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524783    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359a82d1-a799-4f64-ac60-2ddd85732fc3" containerName="maas-api"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524792    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524797    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524806    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524810    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524816    5031 cpu_manager.go:414] "containerMap: removing container" podUID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" containerName="payload-pre-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524821    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" containerName="payload-pre-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524830    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524834    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524842    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" containerName="payload-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524847    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" containerName="payload-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524853    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524869    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing"
Nov 01 23:17:10.524870 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524879    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524884    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524891    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524896    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524902    5031 cpu_manager.go:414] "containerMap: removing container" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524907    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524915    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524920    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524930    5031 cpu_manager.go:414] "containerMap: removing container" podUID="52c8b445-4220-4796-8df4-bfc9b30bd951" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524935    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c8b445-4220-4796-8df4-bfc9b30bd951" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524942    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524947    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524953    5031 cpu_manager.go:414] "containerMap: removing container" podUID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524957    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524962    5031 cpu_manager.go:414] "containerMap: removing container" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524966    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524972    5031 cpu_manager.go:414] "containerMap: removing container" podUID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524976    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524982    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524986    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524993    5031 cpu_manager.go:414] "containerMap: removing container" podUID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.524997    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525003    5031 cpu_manager.go:414] "containerMap: removing container" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525008    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525013    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525017    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525023    5031 cpu_manager.go:414] "containerMap: removing container" podUID="612ecb6b-f262-4723-b71a-3c77ffc45938" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525027    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612ecb6b-f262-4723-b71a-3c77ffc45938" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525034    5031 cpu_manager.go:414] "containerMap: removing container" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525038    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525047    5031 cpu_manager.go:414] "containerMap: removing container" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525051    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525063    5031 cpu_manager.go:414] "containerMap: removing container" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525067    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525074    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525078    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525085    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525090    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525096    5031 cpu_manager.go:414] "containerMap: removing container" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525101    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525150    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6c6195-bd41-45b2-8ece-379c55f3f0a7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525160    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aed3a7-7bfd-4c62-a636-df97ceb3211a" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525165    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525171    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5038f7-f7e4-41e9-94c0-d7fc54cc80ea" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525178    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c8b445-4220-4796-8df4-bfc9b30bd951" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525182    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e6e810-2bcf-4269-9106-7cf61f233eb7" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525187    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d934bf-46d8-4ec1-92d9-3ba4c7fb032b" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525193    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aec9d0-70f3-4823-8fa8-bce08fc89c8b" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525198    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bcb313-b6e0-4fe2-a82e-900e3f715dc6" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525202    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0ff0e9-cf70-42a2-a30b-d0d26adb3ba8" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525207    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c5c865-ba7b-484d-9550-d1d181ff6ef4" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525212    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8d188a-d423-46b7-8370-efa2eb45ba5e" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525218    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="612ecb6b-f262-4723-b71a-3c77ffc45938" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525222    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525227    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabeed4b-35e8-4266-88c1-6bbec4c22cd8" containerName="discovery"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525233    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525238    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea484b4f-95ce-499c-9c25-1ec54cc3967f" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525244    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21812fc-97a7-4b8d-9f2e-ecb49e5a7e49" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525249    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abdde21-b92d-44bf-8454-b346bf136a24" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525252    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e3c20d-d3a2-46d3-98c6-56ca0017a485" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525257    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525263    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525268    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="359a82d1-a799-4f64-ac60-2ddd85732fc3" containerName="maas-api"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525272    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b276892-0640-470f-ba01-70d5f06015ad" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525277    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525283    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7743905f-5888-4188-8ace-fd72d94cdec3" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525287    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb8f7ba-375f-480d-8e7b-b8238cde6934" containerName="payload-pre-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525293    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4580fb-35f4-4d7f-afcc-a00232349ce8" containerName="main"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525299    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45d2f1a-36c0-4f86-bd72-50010f398ac7" containerName="payload-processing"
Nov 01 23:17:10.525456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.525303    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c305d46-d28a-4b6b-90eb-9663bd155d17" containerName="payload-pre-processing"
Nov 01 23:17:10.528355 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.528336    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.589692 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589658    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.589692 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589693    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.589991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589715    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.589991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589731    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.589991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589950    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.590143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.589993    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.590143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.590082    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.590241 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.590142    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwb7\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-kube-api-access-6pwb7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.590241 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.590188    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.627840 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:17:10.627796    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:17:10.690987 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.690939    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.690987 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.690988    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691013    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691049    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwb7\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-kube-api-access-6pwb7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691088    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691111    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691136    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691170    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691279 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691195    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691469    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691504    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.691756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691674    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.692083 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691901    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.692083 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.691998    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.693673 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.693651    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.693855 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.693840    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.739567 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.739514    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"]
Nov 01 23:17:10.745395 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.745323    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.745613 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.745431    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwb7\" (UniqueName: \"kubernetes.io/projected/8e04541d-5090-4ab5-b89a-1d04d3c0a148-kube-api-access-6pwb7\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5zmng\" (UID: \"8e04541d-5090-4ab5-b89a-1d04d3c0a148\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:10.841006 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:10.840964    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:11.264318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:11.264235    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"]
Nov 01 23:17:11.270470 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:17:11.270436    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e04541d_5090_4ab5_b89a_1d04d3c0a148.slice/crio-67b072bae652067cc107ce470ed6edfc70413f6f5bd0fe3b1998a993c1132598 WatchSource:0}: Error finding container 67b072bae652067cc107ce470ed6edfc70413f6f5bd0fe3b1998a993c1132598: Status 404 returned error can't find the container with id 67b072bae652067cc107ce470ed6edfc70413f6f5bd0fe3b1998a993c1132598
Nov 01 23:17:11.272825 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:11.272788    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:11.272945 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:11.272859    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:11.272945 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:11.272889    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:12.250408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.250367    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" event={"ID":"8e04541d-5090-4ab5-b89a-1d04d3c0a148","Type":"ContainerStarted","Data":"dd6601a32f17302949bbf2c14ebc4e7dd336e454f4c5ac8154bc7f220e46d783"}
Nov 01 23:17:12.250951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.250432    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" event={"ID":"8e04541d-5090-4ab5-b89a-1d04d3c0a148","Type":"ContainerStarted","Data":"67b072bae652067cc107ce470ed6edfc70413f6f5bd0fe3b1998a993c1132598"}
Nov 01 23:17:12.450823 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.450765    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podStartSLOduration=2.450750226 podStartE2EDuration="2.450750226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:12.404869407 +0000 UTC m=+3186.360909250" watchObservedRunningTime="2026-11-01 23:17:12.450750226 +0000 UTC m=+3186.406790069"
Nov 01 23:17:12.841397 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.841353    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:12.842866 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.842840    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:12.842973 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:12.842892    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:13.842292 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:13.842251    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:13.842766 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:13.842310    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:14.841975 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:14.841933    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:14.842181 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:14.841989    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:15.841368 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:15.841326    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:15.841815 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:15.841378    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:16.842019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:16.841985    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:16.842527 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:16.842040    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:17.841755 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:17.841712    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5zmng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 01 23:17:17.841939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:17.841776    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng" podUID="8e04541d-5090-4ab5-b89a-1d04d3c0a148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 01 23:17:19.607597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:19.607532    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:19.608125 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:19.607923    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:19.608834 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:19.608813    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5zmng"
Nov 01 23:17:25.625401 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625365    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"]
Nov 01 23:17:25.625852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625734    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:25.625852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625745    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc855f75-bd1e-4b7b-8910-31c3cff3a1c5" containerName="istio-proxy"
Nov 01 23:17:25.625852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625758    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:25.625852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625763    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a203e67-a35d-4644-a6ac-ba1d1de22534" containerName="istio-proxy"
Nov 01 23:17:25.625852 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.625835    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="790c8db7-f00d-4ee9-9686-5b16f2c10d5e" containerName="istio-proxy"
Nov 01 23:17:25.629113 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.629089    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.734394 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.734353    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"]
Nov 01 23:17:25.736207 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736181    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9e3876-1e29-4e71-af35-638422d47e53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736346 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736225    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736346 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736335    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736432 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736377    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmfp\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-kube-api-access-2nmfp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736430    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9e3876-1e29-4e71-af35-638422d47e53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736491 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736455    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736620 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736524    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736658 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736636    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.736726 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.736713    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838272 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838238    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9e3876-1e29-4e71-af35-638422d47e53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838272 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838278    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838388    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838437    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmfp\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-kube-api-access-2nmfp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838465    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9e3876-1e29-4e71-af35-638422d47e53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838503 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838488    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838716 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838512    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838716 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838529    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838716 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838636    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838716 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838678    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.838931 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.838774    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.839149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.839127    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.839273 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.839143    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.839273 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.839257    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9e3876-1e29-4e71-af35-638422d47e53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.840927 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.840902    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9e3876-1e29-4e71-af35-638422d47e53-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.841129 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.841110    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9e3876-1e29-4e71-af35-638422d47e53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.892347 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.892276    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.892953 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.892931    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmfp\" (UniqueName: \"kubernetes.io/projected/bc9e3876-1e29-4e71-af35-638422d47e53-kube-api-access-2nmfp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2zqr6\" (UID: \"bc9e3876-1e29-4e71-af35-638422d47e53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:25.942454 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:25.942403    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:26.367663 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:26.367618    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"]
Nov 01 23:17:26.374885 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:17:26.374856    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9e3876_1e29_4e71_af35_638422d47e53.slice/crio-2a441dfc175f7552e2103701dcfe75bf3b23bcc83e4a57ce3629c40b88ff46ca WatchSource:0}: Error finding container 2a441dfc175f7552e2103701dcfe75bf3b23bcc83e4a57ce3629c40b88ff46ca: Status 404 returned error can't find the container with id 2a441dfc175f7552e2103701dcfe75bf3b23bcc83e4a57ce3629c40b88ff46ca
Nov 01 23:17:26.384688 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:26.384628    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:26.384846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:26.384747    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:26.384846 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:26.384780    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:17:27.348418 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.348373    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" event={"ID":"bc9e3876-1e29-4e71-af35-638422d47e53","Type":"ContainerStarted","Data":"dcc2df8588678c242a0b2d33261e768885b0284e075b701c3a48e301b08ef8cd"}
Nov 01 23:17:27.349009 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.348474    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" event={"ID":"bc9e3876-1e29-4e71-af35-638422d47e53","Type":"ContainerStarted","Data":"2a441dfc175f7552e2103701dcfe75bf3b23bcc83e4a57ce3629c40b88ff46ca"}
Nov 01 23:17:27.550506 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.550447    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podStartSLOduration=2.550430825 podStartE2EDuration="2.550430825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:27.503277977 +0000 UTC m=+3201.459317821" watchObservedRunningTime="2026-11-01 23:17:27.550430825 +0000 UTC m=+3201.506470723"
Nov 01 23:17:27.942772 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.942729    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:27.944629 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.944600    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:27.944777 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:27.944654    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:28.942783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:28.942745    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:28.943192 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:28.942808    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:29.943035 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:29.943001    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:29.943583 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:29.943064    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:30.943758 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:30.943717    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:30.944188 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:30.943776    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:31.943555 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:31.943513    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:31.943760 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:31.943599    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:32.943587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:32.943519    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:32.944000 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:32.943610    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:33.943363 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:33.943325    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 01 23:17:33.943587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:33.943380    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 01 23:17:35.984755 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:35.984709    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:17:35.985317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:35.984785    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:17:36.714141 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:36.714112    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:36.714398 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:36.714377    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:36.715172 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:17:36.715155    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6"
Nov 01 23:17:40.627033 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:17:40.626995    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:18:10.625859 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:18:10.625809    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:18:40.625694 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:18:40.625608    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:19:07.725582 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:19:07.725511    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:19:07.726020 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:19:07.725617    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:19:10.625841 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:19:10.625801    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:19:40.625849 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:19:40.625807    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:20:10.626272 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:20:10.626232    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:20:40.625263 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:20:40.625220    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:21:07.770714 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:07.770665    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2zqr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:21:07.771412 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:07.771373    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2zqr6" podUID="bc9e3876-1e29-4e71-af35-638422d47e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:21:10.626616 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:21:10.626479    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:21:40.624717 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:21:40.624631    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:21:57.754346 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.754308    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:21:57.757953 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.757933    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.806613 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.806581    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3fc07c-openshift-default-dockercfg-msln7\""
Nov 01 23:21:57.863143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.863105    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:21:57.900590 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900516    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.900814 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900622    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.900814 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900647    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.900814 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900790    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.900980 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900836    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.900980 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900941    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.901085 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.900993    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.901152 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.901131    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:57.901209 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:57.901172    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbbj\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002011    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002057    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbbj\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002106    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002128    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002145    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002199    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002226    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002265    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002567    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002582    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002607 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002575    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.002951 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002922    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.003081 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002927    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.003081 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.002933    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.004851 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.004788    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.005044 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.005022    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.057009 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.056969    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbbj\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.057164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.057093    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\") pod \"e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") " pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.071160 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.071131    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:58.498884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.498849    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:21:58.504674 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:21:58.504646    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cffb8d_e35d_4fca_8559_317bb46212c1.slice/crio-37fc5304fcf915664e4dc04f04d1fc72ab6ed958d50c5253e253725101b0429e WatchSource:0}: Error finding container 37fc5304fcf915664e4dc04f04d1fc72ab6ed958d50c5253e253725101b0429e: Status 404 returned error can't find the container with id 37fc5304fcf915664e4dc04f04d1fc72ab6ed958d50c5253e253725101b0429e
Nov 01 23:21:58.506797 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.506763    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:21:58.506878 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.506831    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:21:58.506878 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.506860    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:21:58.774750 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.774650    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerStarted","Data":"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"}
Nov 01 23:21:58.774750 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.774720    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerStarted","Data":"37fc5304fcf915664e4dc04f04d1fc72ab6ed958d50c5253e253725101b0429e"}
Nov 01 23:21:58.884868 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:58.884813    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podStartSLOduration=1.884796943 podStartE2EDuration="1.884796943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:58.88405339 +0000 UTC m=+3472.840093234" watchObservedRunningTime="2026-11-01 23:21:58.884796943 +0000 UTC m=+3472.840836788"
Nov 01 23:21:59.072117 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:59.072081    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:59.077389 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:59.077354    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:59.779954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:59.779917    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:21:59.780947 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:21:59.780923    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:22:05.565680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.565641    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:22:05.569906 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.569877    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.618373 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.618343    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3fc07c\""
Nov 01 23:22:05.664655 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.664619    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c1bca7b9-dockercfg-4dd7p\""
Nov 01 23:22:05.666017 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.665991    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:22:05.679817 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.679777    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.679817 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.679821    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq4c\" (UniqueName: \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.781589 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.781518    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.781788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.781605    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq4c\" (UniqueName: \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.782308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.782286    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.835849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.835781    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq4c\" (UniqueName: \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\") pod \"payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:05.884661 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:05.884625    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:06.254779 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:06.254706    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:22:06.261994 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:22:06.261962    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c9ebc_54ad_47d1_a854_08aafd521d34.slice/crio-74f954dc8053403b974dd3bd1302b18d06667d92b318a5e29414ee42c8ca2e6c WatchSource:0}: Error finding container 74f954dc8053403b974dd3bd1302b18d06667d92b318a5e29414ee42c8ca2e6c: Status 404 returned error can't find the container with id 74f954dc8053403b974dd3bd1302b18d06667d92b318a5e29414ee42c8ca2e6c
Nov 01 23:22:06.814456 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:06.814418    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" event={"ID":"a46c9ebc-54ad-47d1-a854-08aafd521d34","Type":"ContainerStarted","Data":"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"}
Nov 01 23:22:06.814891 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:06.814473    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" event={"ID":"a46c9ebc-54ad-47d1-a854-08aafd521d34","Type":"ContainerStarted","Data":"74f954dc8053403b974dd3bd1302b18d06667d92b318a5e29414ee42c8ca2e6c"}
Nov 01 23:22:06.814891 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:06.814574    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:06.923962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:06.923909    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" podStartSLOduration=1.923895235 podStartE2EDuration="1.923895235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:06.923325981 +0000 UTC m=+3480.879365816" watchObservedRunningTime="2026-11-01 23:22:06.923895235 +0000 UTC m=+3480.879935078"
Nov 01 23:22:10.626296 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:22:10.626253    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:22:17.823373 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:17.823340    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:22:24.928318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.928232    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerID="3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee" exitCode=0
Nov 01 23:22:24.928318 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.928285    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerDied","Data":"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"}
Nov 01 23:22:24.928812 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.928691    5031 scope.go:122] "RemoveContainer" containerID="3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"
Nov 01 23:22:24.930041 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.930005    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:22:24.930147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.930077    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:22:24.930147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:24.930113    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:22:25.935168 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:25.935133    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerStarted","Data":"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"}
Nov 01 23:22:26.075213 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:26.075178    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:22:26.078733 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:26.078698    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:26.078954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:26.078756    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:27.074564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:27.074022    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:27.074564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:27.074089    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:28.073562 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:28.072995    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:28.073562 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:28.073102    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:29.075564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:29.074990    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:29.075564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:29.075056    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:29.780717 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:29.780674    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:22:30.072005 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.071964    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:30.072310 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.072283    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:30.209920 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.209889    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:22:30.213777 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.213752    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.316278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.316246    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3fc07c\"/\"openshift-service-ca.crt\""
Nov 01 23:22:30.316278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.316260    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3fc07c\"/\"kube-root-ca.crt\""
Nov 01 23:22:30.316574 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.316379    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3fc07c\"/\"simulator-single-main-3fc07c-kserve-self-signed-certs\""
Nov 01 23:22:30.316574 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.316472    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3fc07c\"/\"default-dockercfg-n8rnk\""
Nov 01 23:22:30.318151 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.318127    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:22:30.346792 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.346696    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.346792 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.346768    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.347044 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.346914    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.347044 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.347014    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjw8\" (UniqueName: \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.347168 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.347046    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.347168 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.347153    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.448663 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448620    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.448885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448739    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.448885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448769    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.448885 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448795    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjw8\" (UniqueName: \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.449073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448888    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.449073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.448947    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.449189 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.449091    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.449358 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.449338    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.449574 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.449496    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.451404 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.451381    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.451612 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.451593    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.503376 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.503343    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjw8\" (UniqueName: \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\") pod \"simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") " pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.526149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.526108    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:30.930226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.930146    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:22:30.936856 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:22:30.936824    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c61deac_dbcc_45ab_9275_af77eae98764.slice/crio-17445eada571cc8d02bd3c9d2fc04f0fc44f48a0a7e755990eda22ccd5365065 WatchSource:0}: Error finding container 17445eada571cc8d02bd3c9d2fc04f0fc44f48a0a7e755990eda22ccd5365065: Status 404 returned error can't find the container with id 17445eada571cc8d02bd3c9d2fc04f0fc44f48a0a7e755990eda22ccd5365065
Nov 01 23:22:30.970606 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:30.970570    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" event={"ID":"4c61deac-dbcc-45ab-9275-af77eae98764","Type":"ContainerStarted","Data":"17445eada571cc8d02bd3c9d2fc04f0fc44f48a0a7e755990eda22ccd5365065"}
Nov 01 23:22:31.071884 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:31.071841    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 01 23:22:31.072073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:31.071921    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 01 23:22:31.976553 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:31.976501    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" event={"ID":"4c61deac-dbcc-45ab-9275-af77eae98764","Type":"ContainerStarted","Data":"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"}
Nov 01 23:22:32.075907 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:32.075872    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:22:32.076962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:32.076940    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:22:32.083396 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:32.083340    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" podStartSLOduration=2.083323736 podStartE2EDuration="2.083323736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:32.08262617 +0000 UTC m=+3506.038666016" watchObservedRunningTime="2026-11-01 23:22:32.083323736 +0000 UTC m=+3506.039363590"
Nov 01 23:22:32.526827 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:32.526784    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:40.526841 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:40.526798    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:40.539480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:40.539451    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:22:40.625947 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:22:40.625908    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:22:41.033579 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:22:41.033531    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:23:10.625812 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:23:10.625772    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:23:36.091326 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:36.091287    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:23:40.635261 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:23:40.635217    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:23:43.151163 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:43.151125    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:23:48.152246 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:48.152177    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing" containerID="cri-o://f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880" gracePeriod=25
Nov 01 23:23:51.112502 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.112438    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" podUID="4c61deac-dbcc-45ab-9275-af77eae98764" containerName="main" containerID="cri-o://8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a" gracePeriod=45
Nov 01 23:23:51.368200 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.368122    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:23:51.372236 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.372202    5031 generic.go:356] "Generic (PLEG): container finished" podID="4c61deac-dbcc-45ab-9275-af77eae98764" containerID="8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a" exitCode=0
Nov 01 23:23:51.372412 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.372269    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"
Nov 01 23:23:51.372412 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.372270    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" event={"ID":"4c61deac-dbcc-45ab-9275-af77eae98764","Type":"ContainerDied","Data":"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"}
Nov 01 23:23:51.372500 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.372427    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l" event={"ID":"4c61deac-dbcc-45ab-9275-af77eae98764","Type":"ContainerDied","Data":"17445eada571cc8d02bd3c9d2fc04f0fc44f48a0a7e755990eda22ccd5365065"}
Nov 01 23:23:51.372500 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.372457    5031 scope.go:122] "RemoveContainer" containerID="8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"
Nov 01 23:23:51.381201 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.381177    5031 scope.go:122] "RemoveContainer" containerID="8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"
Nov 01 23:23:51.381559 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:23:51.381525    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a\": container with ID starting with 8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a not found: ID does not exist" containerID="8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"
Nov 01 23:23:51.381630 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.381571    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a"} err="failed to get container status \"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a\": rpc error: code = NotFound desc = could not find container \"8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a\": container with ID starting with 8f9073c925f68316ed77675c70783170ea8fb21933a9688a57223bcd50e63e3a not found: ID does not exist"
Nov 01 23:23:51.470246 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470210    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\" (UniqueName: \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470439 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470257    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470439 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470308    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470439 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470414    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470660 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470466    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470660 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470523    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\") pod \"4c61deac-dbcc-45ab-9275-af77eae98764\" (UID: \"4c61deac-dbcc-45ab-9275-af77eae98764\") "
Nov 01 23:23:51.470849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470798    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:23:51.470924 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470832    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:23:51.470924 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470866    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:23:51.471037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470957    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.471037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470971    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.471037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.470982    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.472813 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.472785    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:23:51.472947 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.472788    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:23:51.472947 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.472823    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8" pod "4c61deac-dbcc-45ab-9275-af77eae98764" (UID: "4c61deac-dbcc-45ab-9275-af77eae98764"). InnerVolumeSpecName "kube-api-access-mwjw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:23:51.571909 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.571871    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjw8\" (UniqueName: \"kubernetes.io/projected/4c61deac-dbcc-45ab-9275-af77eae98764-kube-api-access-mwjw8\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.571909 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.571904    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c61deac-dbcc-45ab-9275-af77eae98764-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.571909 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.571913    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c61deac-dbcc-45ab-9275-af77eae98764-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:51.782374 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.782341    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:23:51.787320 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:51.787288    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3fc07c/simulator-single-main-3fc07c-kserve-6868dc45c4-npn7l"]
Nov 01 23:23:52.170593 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:52.170527    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c61deac-dbcc-45ab-9275-af77eae98764" path="/var/lib/kubelet/pods/4c61deac-dbcc-45ab-9275-af77eae98764/volumes"
Nov 01 23:23:57.821508 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:57.821472    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:23:59.140942 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:59.140900    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:23:59.141431 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:23:59.141171    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" containerID="cri-o://eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c" gracePeriod=30
Nov 01 23:24:02.078820 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:02.078778    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:24:02.079229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:02.078841    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:24:04.396889 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.396861    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:24:04.444401 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.444316    5031 generic.go:356] "Generic (PLEG): container finished" podID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerID="eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c" exitCode=0
Nov 01 23:24:04.444401 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.444375    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"
Nov 01 23:24:04.444637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.444401    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerDied","Data":"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"}
Nov 01 23:24:04.444637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.444461    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc" event={"ID":"e4cffb8d-e35d-4fca-8559-317bb46212c1","Type":"ContainerDied","Data":"37fc5304fcf915664e4dc04f04d1fc72ab6ed958d50c5253e253725101b0429e"}
Nov 01 23:24:04.444637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.444487    5031 scope.go:122] "RemoveContainer" containerID="eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"
Nov 01 23:24:04.453316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.453298    5031 scope.go:122] "RemoveContainer" containerID="3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"
Nov 01 23:24:04.461920 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.461894    5031 scope.go:122] "RemoveContainer" containerID="eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"
Nov 01 23:24:04.462256 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:24:04.462235    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c\": container with ID starting with eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c not found: ID does not exist" containerID="eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"
Nov 01 23:24:04.462320 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.462270    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c"} err="failed to get container status \"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c\": rpc error: code = NotFound desc = could not find container \"eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c\": container with ID starting with eb3427b3c44466a9062399a28b829776e2b26a2b13e93f4c5326e40ceb5a967c not found: ID does not exist"
Nov 01 23:24:04.462320 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.462289    5031 scope.go:122] "RemoveContainer" containerID="3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"
Nov 01 23:24:04.462596 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:24:04.462576    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee\": container with ID starting with 3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee not found: ID does not exist" containerID="3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"
Nov 01 23:24:04.462653 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.462604    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee"} err="failed to get container status \"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee\": rpc error: code = NotFound desc = could not find container \"3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee\": container with ID starting with 3aa346276c4b361003ec088c731413b356c09206f7386a0e556edd4162441dee not found: ID does not exist"
Nov 01 23:24:04.500822 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.500781    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.500848    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.500917    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.500947    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501030 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501003    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501057    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501090    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501099    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501140    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501177    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\") pod \"e4cffb8d-e35d-4fca-8559-317bb46212c1\" (UID: \"e4cffb8d-e35d-4fca-8559-317bb46212c1\") "
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501194    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:04.501266 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501222    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:04.501649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501334    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:04.501649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501529    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:24:04.501649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501612    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.501649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501639    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.501812 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501656    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.501812 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.501670    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.503482 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.503453    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:24:04.503682 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.503504    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "kube-api-access-twbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:04.503898 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.503880    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:04.503962 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.503939    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy" pod "e4cffb8d-e35d-4fca-8559-317bb46212c1" (UID: "e4cffb8d-e35d-4fca-8559-317bb46212c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:04.602196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.602159    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.602196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.602193    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbbj\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-kube-api-access-twbbj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.602196 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.602205    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cffb8d-e35d-4fca-8559-317bb46212c1-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.602451 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.602214    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.602451 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.602222    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cffb8d-e35d-4fca-8559-317bb46212c1-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:04.858963 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.858926    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:24:04.865692 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:04.865660    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3fc07c-openshift-default-76ff577749qt2gc"]
Nov 01 23:24:06.170133 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:06.170099    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" path="/var/lib/kubelet/pods/e4cffb8d-e35d-4fca-8559-317bb46212c1/volumes"
Nov 01 23:24:07.821616 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:07.821568    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:24:08.416429 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.416354    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:24:08.465685 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.465651    5031 generic.go:356] "Generic (PLEG): container finished" podID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerID="f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880" exitCode=0
Nov 01 23:24:08.465864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.465701    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"
Nov 01 23:24:08.465864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.465712    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" event={"ID":"a46c9ebc-54ad-47d1-a854-08aafd521d34","Type":"ContainerDied","Data":"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"}
Nov 01 23:24:08.465864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.465782    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg" event={"ID":"a46c9ebc-54ad-47d1-a854-08aafd521d34","Type":"ContainerDied","Data":"74f954dc8053403b974dd3bd1302b18d06667d92b318a5e29414ee42c8ca2e6c"}
Nov 01 23:24:08.465864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.465809    5031 scope.go:122] "RemoveContainer" containerID="f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"
Nov 01 23:24:08.474193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.474168    5031 scope.go:122] "RemoveContainer" containerID="f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"
Nov 01 23:24:08.474509 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:24:08.474491    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880\": container with ID starting with f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880 not found: ID does not exist" containerID="f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"
Nov 01 23:24:08.474588 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.474519    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880"} err="failed to get container status \"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880\": rpc error: code = NotFound desc = could not find container \"f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880\": container with ID starting with f3924899d7beb053a8d0130af56191dd01a21177f0bb79b490fedf2850c16880 not found: ID does not exist"
Nov 01 23:24:08.537864 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.537831    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\") pod \"a46c9ebc-54ad-47d1-a854-08aafd521d34\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") "
Nov 01 23:24:08.538074 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.537890    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\" (UniqueName: \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\") pod \"a46c9ebc-54ad-47d1-a854-08aafd521d34\" (UID: \"a46c9ebc-54ad-47d1-a854-08aafd521d34\") "
Nov 01 23:24:08.538299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.538268    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume" pod "a46c9ebc-54ad-47d1-a854-08aafd521d34" (UID: "a46c9ebc-54ad-47d1-a854-08aafd521d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:24:08.540243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.540216    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c" pod "a46c9ebc-54ad-47d1-a854-08aafd521d34" (UID: "a46c9ebc-54ad-47d1-a854-08aafd521d34"). InnerVolumeSpecName "kube-api-access-hdq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:08.639499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.639445    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a46c9ebc-54ad-47d1-a854-08aafd521d34-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:08.639499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.639491    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdq4c\" (UniqueName: \"kubernetes.io/projected/a46c9ebc-54ad-47d1-a854-08aafd521d34-kube-api-access-hdq4c\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:08.874564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.874503    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:24:08.877922 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:08.877890    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3fc07c-7c977745cd8hzg"]
Nov 01 23:24:10.169731 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:24:10.169695    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" path="/var/lib/kubelet/pods/a46c9ebc-54ad-47d1-a854-08aafd521d34/volumes"
Nov 01 23:24:10.625637 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:24:10.625603    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:24:40.627369 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:24:40.627328    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:25:10.626075 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:25:10.626033    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:25:40.626409 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:25:40.626367    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:26:10.625506 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:26:10.625458    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:26:30.789054 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789023    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789366    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789377    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789392    5031 cpu_manager.go:414] "containerMap: removing container" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789397    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789407    5031 cpu_manager.go:414] "containerMap: removing container" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789412    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789433    5031 cpu_manager.go:414] "containerMap: removing container" podUID="4c61deac-dbcc-45ab-9275-af77eae98764" containerName="main"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789438    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c61deac-dbcc-45ab-9275-af77eae98764" containerName="main"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789505    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c61deac-dbcc-45ab-9275-af77eae98764" containerName="main"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789514    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789522    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46c9ebc-54ad-47d1-a854-08aafd521d34" containerName="payload-processing"
Nov 01 23:26:30.789597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.789528    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cffb8d-e35d-4fca-8559-317bb46212c1" containerName="istio-proxy"
Nov 01 23:26:30.792791 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.792770    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.841313 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.841277    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-0cff29-openshift-default-dockercfg-qb67b\""
Nov 01 23:26:30.897804 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.897772    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:26:30.961675 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961631    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961867 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961699    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961867 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961790    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961867 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961833    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961899    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961932    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.961979 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.961970    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.962107 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.962092    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7nj\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:30.962144 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:30.962117    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062675 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062643    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062692    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062735    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062759    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062807    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.062877 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062874    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7nj\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063170 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062902    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063170 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.062937    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063170 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063004    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063170 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063134    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063392 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063189    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063392 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063243    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063498 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063418    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.063680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.063661    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.065277 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.065257    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.065634 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.065616    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.117834 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.117797    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.118014 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.117944    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7nj\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\") pod \"e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") " pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.405775 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.405689    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:31.827871 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.827832    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:26:31.833813 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:26:31.833780    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c512156_80a9_4188_8cd7_937d9d23a4a5.slice/crio-2bdd8ad67bde97ff6593209e3a37b7aec00d2502de55f45b297425c56df049ba WatchSource:0}: Error finding container 2bdd8ad67bde97ff6593209e3a37b7aec00d2502de55f45b297425c56df049ba: Status 404 returned error can't find the container with id 2bdd8ad67bde97ff6593209e3a37b7aec00d2502de55f45b297425c56df049ba
Nov 01 23:26:31.836046 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.836010    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:26:31.836151 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.836079    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:26:31.836151 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:31.836109    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:26:32.141261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:32.141180    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerStarted","Data":"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"}
Nov 01 23:26:32.141261 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:32.141233    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerStarted","Data":"2bdd8ad67bde97ff6593209e3a37b7aec00d2502de55f45b297425c56df049ba"}
Nov 01 23:26:32.251208 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:32.251142    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podStartSLOduration=2.251129301 podStartE2EDuration="2.251129301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:26:32.250470561 +0000 UTC m=+3746.206510405" watchObservedRunningTime="2026-11-01 23:26:32.251129301 +0000 UTC m=+3746.207169145"
Nov 01 23:26:32.406653 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:32.406570    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:32.412008 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:32.411980    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:33.146389 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:33.146355    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:33.147260 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:33.147243    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:26:38.937287 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:38.937252    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:26:38.941184 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:38.941146    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:38.989531 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:38.989491    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-0cff29\""
Nov 01 23:26:39.035770 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.035736    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cb26499d-dockercfg-b8fsq\""
Nov 01 23:26:39.037177 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.037153    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:26:39.138434 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.138396    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvtj\" (UniqueName: \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.138664 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.138465    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.239436 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.239343    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvtj\" (UniqueName: \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.239436 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.239417    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.240144 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.240118    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.292958 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.292925    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvtj\" (UniqueName: \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\") pod \"payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.555469 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.555431    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:39.926416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:39.926327    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:26:39.932553 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:26:39.932496    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fb8fcc_3571_4d82_bbc3_4f8b6756ef76.slice/crio-aa6ab4304abcccbeefa9e33e922995ec9e3ec62d9791cf8f12c4c72e016364c1 WatchSource:0}: Error finding container aa6ab4304abcccbeefa9e33e922995ec9e3ec62d9791cf8f12c4c72e016364c1: Status 404 returned error can't find the container with id aa6ab4304abcccbeefa9e33e922995ec9e3ec62d9791cf8f12c4c72e016364c1
Nov 01 23:26:40.180094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:40.179996    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" event={"ID":"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76","Type":"ContainerStarted","Data":"156508bf4c263a2f022b9de6d171ed84929b403ec4fcfdb1ca2d7a38950f93bc"}
Nov 01 23:26:40.180094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:40.180055    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" event={"ID":"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76","Type":"ContainerStarted","Data":"aa6ab4304abcccbeefa9e33e922995ec9e3ec62d9791cf8f12c4c72e016364c1"}
Nov 01 23:26:40.180814 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:40.180123    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:40.291619 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:40.291557    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" podStartSLOduration=2.291521861 podStartE2EDuration="2.291521861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:26:40.29037522 +0000 UTC m=+3754.246415064" watchObservedRunningTime="2026-11-01 23:26:40.291521861 +0000 UTC m=+3754.247561704"
Nov 01 23:26:40.626531 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:26:40.626487    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:26:51.187939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:51.187909    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:26:59.267102 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.267068    5031 generic.go:356] "Generic (PLEG): container finished" podID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerID="4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788" exitCode=0
Nov 01 23:26:59.267589 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.267142    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerDied","Data":"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"}
Nov 01 23:26:59.267638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.267607    5031 scope.go:122] "RemoveContainer" containerID="4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"
Nov 01 23:26:59.268883 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.268849    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:26:59.269004 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.268944    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:26:59.269004 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:26:59.268991    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:27:00.282834 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:00.282794    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerStarted","Data":"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"}
Nov 01 23:27:00.406299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:00.406258    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:27:00.407451 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:00.407428    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 23:27:00.407556 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:00.407478    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 23:27:01.406927 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:01.406894    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 23:27:01.407360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:01.406956    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 23:27:02.406271 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:02.406231    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 23:27:02.406570 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:02.406304    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 23:27:02.941942 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:02.941902    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:27:02.945359 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:02.945339    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:02.994043 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:02.993996    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0cff29\"/\"openshift-service-ca.crt\""
Nov 01 23:27:03.040028 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.039996    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-0cff29\"/\"kube-root-ca.crt\""
Nov 01 23:27:03.040028 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.040018    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0cff29\"/\"default-dockercfg-6wlxd\""
Nov 01 23:27:03.040263 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.040018    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-0cff29\"/\"simulator-single-main-0cff29-kserve-self-signed-certs\""
Nov 01 23:27:03.041683 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.041649    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:27:03.113289 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113255    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.113533 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113320    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.113533 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113413    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6st\" (UniqueName: \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.113533 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113521    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.113701 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113592    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.113701 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.113621    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.147250 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.147213    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:27:03.215229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215139    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215177    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215206    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215465 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215336    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215465 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215439    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215581 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215490    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6st\" (UniqueName: \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215640    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215743 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215692    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.215800 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.215787    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.217840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.217808    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.218053 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.218031    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.278568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.278490    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6st\" (UniqueName: \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\") pod \"simulator-single-main-0cff29-kserve-5788b9fdb-sshtp\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") " pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.406280 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.406241    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 23:27:03.406502 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.406304    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 23:27:03.571829 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.571764    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:03.949165 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:03.949081    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:27:03.980147 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:27:03.980096    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe9c5c65_6b1d_4828_bcc3_1a518a1e5699.slice/crio-2e093c6b9654e78ffe268313468c2f650dc492bc788832e417d53c85af553742 WatchSource:0}: Error finding container 2e093c6b9654e78ffe268313468c2f650dc492bc788832e417d53c85af553742: Status 404 returned error can't find the container with id 2e093c6b9654e78ffe268313468c2f650dc492bc788832e417d53c85af553742
Nov 01 23:27:04.302344 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.302310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" event={"ID":"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699","Type":"ContainerStarted","Data":"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"}
Nov 01 23:27:04.302589 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.302354    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" event={"ID":"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699","Type":"ContainerStarted","Data":"2e093c6b9654e78ffe268313468c2f650dc492bc788832e417d53c85af553742"}
Nov 01 23:27:04.406523 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.406490    5031 patch_prober.go:29] interesting pod/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 01 23:27:04.406726 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.406555    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 01 23:27:04.433690 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.433628    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" podStartSLOduration=2.433614316 podStartE2EDuration="2.433614316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:27:04.432299813 +0000 UTC m=+3778.388339656" watchObservedRunningTime="2026-11-01 23:27:04.433614316 +0000 UTC m=+3778.389654157"
Nov 01 23:27:04.573745 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:04.573635    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:05.504391 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:05.504338    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:27:05.506373 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:05.506347    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:27:10.625496 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:27:10.625461    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:27:13.573892 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:13.573847    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:13.586697 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:13.586667    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:14.360437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:27:14.360401    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:27:40.626197 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:27:40.626116    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:28:04.611565 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:04.611507    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:28:10.627084 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:28:10.627048    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:28:11.626818 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:11.626772    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:28:16.627269 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:16.627214    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing" containerID="cri-o://156508bf4c263a2f022b9de6d171ed84929b403ec4fcfdb1ca2d7a38950f93bc" gracePeriod=25
Nov 01 23:28:19.632724 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:19.632677    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" podUID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" containerName="main" containerID="cri-o://8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4" gracePeriod=45
Nov 01 23:28:19.889069 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:19.888996    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:28:20.006765 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.006711    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.006765 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.006778    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.007063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.006824    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.007063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.006846    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\" (UniqueName: \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.007063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.006865    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.007063 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007024    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\") pod \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\" (UID: \"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699\") "
Nov 01 23:28:20.007282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007087    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:20.007282 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007172    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:20.007391 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007277    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:20.007452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007420    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.007452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007435    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.007452 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.007445    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.009254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.009230    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:20.009373 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.009257    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:28:20.009373 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.009301    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st" pod "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" (UID: "fe9c5c65-6b1d-4828-bcc3-1a518a1e5699"). InnerVolumeSpecName "kube-api-access-6l6st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:20.108019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.107986    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.108019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.108017    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.108227 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.108029    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6st\" (UniqueName: \"kubernetes.io/projected/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699-kube-api-access-6l6st\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:20.677930 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.677892    5031 generic.go:356] "Generic (PLEG): container finished" podID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" containerID="8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4" exitCode=0
Nov 01 23:28:20.678446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.677951    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" event={"ID":"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699","Type":"ContainerDied","Data":"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"}
Nov 01 23:28:20.678446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.677980    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp" event={"ID":"fe9c5c65-6b1d-4828-bcc3-1a518a1e5699","Type":"ContainerDied","Data":"2e093c6b9654e78ffe268313468c2f650dc492bc788832e417d53c85af553742"}
Nov 01 23:28:20.678446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.677965    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"
Nov 01 23:28:20.678446 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.677996    5031 scope.go:122] "RemoveContainer" containerID="8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"
Nov 01 23:28:20.686643 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.686617    5031 scope.go:122] "RemoveContainer" containerID="8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"
Nov 01 23:28:20.686979 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:28:20.686954    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4\": container with ID starting with 8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4 not found: ID does not exist" containerID="8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"
Nov 01 23:28:20.687089 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.686985    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4"} err="failed to get container status \"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4\": rpc error: code = NotFound desc = could not find container \"8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4\": container with ID starting with 8c571ca82cecb06246a03b7583b50325ab26692897a5a535a42f45fb8ca549d4 not found: ID does not exist"
Nov 01 23:28:20.830564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.830510    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:28:20.876398 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:20.876364    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-0cff29/simulator-single-main-0cff29-kserve-5788b9fdb-sshtp"]
Nov 01 23:28:21.185902 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:21.185849    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:28:22.169511 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:22.169470    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" path="/var/lib/kubelet/pods/fe9c5c65-6b1d-4828-bcc3-1a518a1e5699/volumes"
Nov 01 23:28:26.367345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:26.367309    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:28:26.367891 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:26.367588    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy" containerID="cri-o://a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0" gracePeriod=30
Nov 01 23:28:31.186170 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.186122    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:28:31.639484 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.639456    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:28:31.727967 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.727938    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728120 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.727992    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728120 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728067    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728120 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728096    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728124    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728157    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728225    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728254    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:31.728300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728291    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728332    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\") pod \"8c512156-80a9-4188-8cd7-937d9d23a4a5\" (UID: \"8c512156-80a9-4188-8cd7-937d9d23a4a5\") "
Nov 01 23:28:31.728604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728418    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:31.728715 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728632    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:31.728768 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728756    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.728897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728776    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.728897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728792    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.728897 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.728873    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:31.729134 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.729111    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:28:31.730672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.730633    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:28:31.730832 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.730810    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:28:31.731060 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.731038    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "kube-api-access-kj7nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:31.731163 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.731066    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token" pod "8c512156-80a9-4188-8cd7-937d9d23a4a5" (UID: "8c512156-80a9-4188-8cd7-937d9d23a4a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:31.733805 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.733777    5031 generic.go:356] "Generic (PLEG): container finished" podID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerID="a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0" exitCode=0
Nov 01 23:28:31.733908 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.733844    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"
Nov 01 23:28:31.733966 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.733928    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerDied","Data":"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"}
Nov 01 23:28:31.734000 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.733977    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd" event={"ID":"8c512156-80a9-4188-8cd7-937d9d23a4a5","Type":"ContainerDied","Data":"2bdd8ad67bde97ff6593209e3a37b7aec00d2502de55f45b297425c56df049ba"}
Nov 01 23:28:31.734032 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.734011    5031 scope.go:122] "RemoveContainer" containerID="a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"
Nov 01 23:28:31.751166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.751141    5031 scope.go:122] "RemoveContainer" containerID="4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"
Nov 01 23:28:31.759811 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.759786    5031 scope.go:122] "RemoveContainer" containerID="a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"
Nov 01 23:28:31.760171 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:28:31.760150    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0\": container with ID starting with a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0 not found: ID does not exist" containerID="a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"
Nov 01 23:28:31.760226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.760181    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0"} err="failed to get container status \"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0\": rpc error: code = NotFound desc = could not find container \"a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0\": container with ID starting with a41166775dd94cf189bb678aeddcc26dc5d34113bb4bbd622236016d14f4d6e0 not found: ID does not exist"
Nov 01 23:28:31.760226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.760202    5031 scope.go:122] "RemoveContainer" containerID="4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"
Nov 01 23:28:31.760499 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:28:31.760474    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788\": container with ID starting with 4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788 not found: ID does not exist" containerID="4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"
Nov 01 23:28:31.760602 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.760511    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788"} err="failed to get container status \"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788\": rpc error: code = NotFound desc = could not find container \"4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788\": container with ID starting with 4eee29de556e0b76124ef02b3910c0dff7d2feb7de45f001d5564b22b93d3788 not found: ID does not exist"
Nov 01 23:28:31.830278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830234    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.830278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830272    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.830278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830282    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c512156-80a9-4188-8cd7-937d9d23a4a5-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.830521 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830294    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c512156-80a9-4188-8cd7-937d9d23a4a5-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.830521 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830304    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj7nj\" (UniqueName: \"kubernetes.io/projected/8c512156-80a9-4188-8cd7-937d9d23a4a5-kube-api-access-kj7nj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.830521 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.830313    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c512156-80a9-4188-8cd7-937d9d23a4a5-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:31.897635 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.897597    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:28:31.901190 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:31.901162    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-0cff29-openshift-default-94f8f5d6f-4b2jd"]
Nov 01 23:28:32.169861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.169822    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" path="/var/lib/kubelet/pods/8c512156-80a9-4188-8cd7-937d9d23a4a5/volumes"
Nov 01 23:28:32.747772 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.747735    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:28:32.748193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748126    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:28:32.748193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748139    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:28:32.748193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748180    5031 cpu_manager.go:414] "containerMap: removing container" podUID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" containerName="main"
Nov 01 23:28:32.748193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748187    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" containerName="main"
Nov 01 23:28:32.748327 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748268    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:28:32.748327 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.748283    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9c5c65-6b1d-4828-bcc3-1a518a1e5699" containerName="main"
Nov 01 23:28:32.753321 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.753299    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840384 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840342    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840618 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840411    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840618 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840507    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840618 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840604    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt8w\" (UniqueName: \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840684    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.840776 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.840715    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.849460 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.849425    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-863b60f0-kserve-self-signed-certs\""
Nov 01 23:28:32.851004 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.850978    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:28:32.942094 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942057    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942103    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942124    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt8w\" (UniqueName: \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942178    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942206    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942296    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942663 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942589    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942663 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942633    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.942730 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.942668    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.944904 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.944880    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.945070 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.945052    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:32.995839 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:32.995806    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt8w\" (UniqueName: \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\") pod \"e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") " pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:33.065164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:33.065126    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:33.436680 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:33.436603    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:28:33.442626 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:28:33.442599    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod357ec568_161a_41bb_bede_1eb9658d6461.slice/crio-6e9626de32bcd7bc375d0bc8833c75f795a1ab565b300b8e20074123b87f9598 WatchSource:0}: Error finding container 6e9626de32bcd7bc375d0bc8833c75f795a1ab565b300b8e20074123b87f9598: Status 404 returned error can't find the container with id 6e9626de32bcd7bc375d0bc8833c75f795a1ab565b300b8e20074123b87f9598
Nov 01 23:28:33.747025 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:33.746932    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" event={"ID":"357ec568-161a-41bb-bede-1eb9658d6461","Type":"ContainerStarted","Data":"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"}
Nov 01 23:28:33.747025 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:33.746979    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" event={"ID":"357ec568-161a-41bb-bede-1eb9658d6461","Type":"ContainerStarted","Data":"6e9626de32bcd7bc375d0bc8833c75f795a1ab565b300b8e20074123b87f9598"}
Nov 01 23:28:33.855639 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:33.855574    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" podStartSLOduration=1.855556193 podStartE2EDuration="1.855556193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:28:33.852940093 +0000 UTC m=+3867.808979937" watchObservedRunningTime="2026-11-01 23:28:33.855556193 +0000 UTC m=+3867.811596034"
Nov 01 23:28:34.065974 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:34.065939    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:36.761948 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.761905    5031 generic.go:356] "Generic (PLEG): container finished" podID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerID="156508bf4c263a2f022b9de6d171ed84929b403ec4fcfdb1ca2d7a38950f93bc" exitCode=0
Nov 01 23:28:36.762469 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.761968    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" event={"ID":"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76","Type":"ContainerDied","Data":"156508bf4c263a2f022b9de6d171ed84929b403ec4fcfdb1ca2d7a38950f93bc"}
Nov 01 23:28:36.884014 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.883946    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:28:36.981934 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.981896    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\" (UniqueName: \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\") pod \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") "
Nov 01 23:28:36.982134 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.981980    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\") pod \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\" (UID: \"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76\") "
Nov 01 23:28:36.982397 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.982374    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume" pod "f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" (UID: "f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:28:36.984295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:36.984272    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj" pod "f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" (UID: "f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76"). InnerVolumeSpecName "kube-api-access-dhvtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:37.083298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.083259    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvtj\" (UniqueName: \"kubernetes.io/projected/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-kube-api-access-dhvtj\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:37.083298 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.083291    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:37.769244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.769198    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx" event={"ID":"f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76","Type":"ContainerDied","Data":"aa6ab4304abcccbeefa9e33e922995ec9e3ec62d9791cf8f12c4c72e016364c1"}
Nov 01 23:28:37.769244 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.769233    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"
Nov 01 23:28:37.769806 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.769281    5031 scope.go:122] "RemoveContainer" containerID="156508bf4c263a2f022b9de6d171ed84929b403ec4fcfdb1ca2d7a38950f93bc"
Nov 01 23:28:37.921435 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.921393    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:28:37.967410 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:37.967369    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-0cff29-5cc8c4df549dxx"]
Nov 01 23:28:38.171866 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:38.171834    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" path="/var/lib/kubelet/pods/f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76/volumes"
Nov 01 23:28:39.623756 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:39.623695    5031 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fx6mf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:28:39.624455 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:39.624421    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fx6mf" podUID="e35dfd80-6c4d-47df-9581-698a23fe2ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:28:40.626155 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:28:40.626111    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:28:43.066218 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:43.066169    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:43.078704 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:43.078676    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:28:43.835020 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:28:43.834988    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:29:10.625816 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:29:10.625776    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:29:40.626304 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:29:40.626262    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:30:02.483778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:02.483741    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:30:10.625867 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:30:10.625825    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:30:17.504924 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.504819    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" podUID="357ec568-161a-41bb-bede-1eb9658d6461" containerName="main" containerID="cri-o://f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6" gracePeriod=45
Nov 01 23:30:17.763530 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.763453    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:30:17.949360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949319    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\" (UniqueName: \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949578 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949374    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949578 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949420    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949578 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949450    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\" (UniqueName: \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949578 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949475    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949578 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949496    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\") pod \"357ec568-161a-41bb-bede-1eb9658d6461\" (UID: \"357ec568-161a-41bb-bede-1eb9658d6461\") "
Nov 01 23:30:17.949895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949782    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:17.949895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949798    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:17.949895 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949807    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:17.950058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949914    5031 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-home\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:17.950058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949936    5031 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-tmp-dir\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:17.950058 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.949949    5031 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-model-cache\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:17.951883 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.951837    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:30:17.952010 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.951931    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:30:17.952056 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:17.952013    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w" pod "357ec568-161a-41bb-bede-1eb9658d6461" (UID: "357ec568-161a-41bb-bede-1eb9658d6461"). InnerVolumeSpecName "kube-api-access-sxt8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:30:18.051642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.051592    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxt8w\" (UniqueName: \"kubernetes.io/projected/357ec568-161a-41bb-bede-1eb9658d6461-kube-api-access-sxt8w\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:18.051642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.051635    5031 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357ec568-161a-41bb-bede-1eb9658d6461-dshm\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:18.051642 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.051650    5031 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357ec568-161a-41bb-bede-1eb9658d6461-tls-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:18.371420 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.371328    5031 generic.go:356] "Generic (PLEG): container finished" podID="357ec568-161a-41bb-bede-1eb9658d6461" containerID="f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6" exitCode=0
Nov 01 23:30:18.371420 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.371375    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"
Nov 01 23:30:18.371420 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.371389    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" event={"ID":"357ec568-161a-41bb-bede-1eb9658d6461","Type":"ContainerDied","Data":"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"}
Nov 01 23:30:18.371721 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.371431    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm" event={"ID":"357ec568-161a-41bb-bede-1eb9658d6461","Type":"ContainerDied","Data":"6e9626de32bcd7bc375d0bc8833c75f795a1ab565b300b8e20074123b87f9598"}
Nov 01 23:30:18.371721 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.371452    5031 scope.go:122] "RemoveContainer" containerID="f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"
Nov 01 23:30:18.381693 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.381672    5031 scope.go:122] "RemoveContainer" containerID="f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"
Nov 01 23:30:18.382073 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:30:18.382053    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6\": container with ID starting with f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6 not found: ID does not exist" containerID="f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"
Nov 01 23:30:18.382166 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.382084    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6"} err="failed to get container status \"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6\": rpc error: code = NotFound desc = could not find container \"f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6\": container with ID starting with f2d52b6a1c1d0a76dc4813766a49fa55dba072dcebdcba88f2ff2714d92c1eb6 not found: ID does not exist"
Nov 01 23:30:18.548808 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.548772    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:30:18.594532 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:18.594490    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-863b60f0-kserve-56d6cbd6d9-dgxgm"]
Nov 01 23:30:20.170635 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:20.170594    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357ec568-161a-41bb-bede-1eb9658d6461" path="/var/lib/kubelet/pods/357ec568-161a-41bb-bede-1eb9658d6461/volumes"
Nov 01 23:30:40.626444 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:30:40.626405    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:30:53.901450 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.901402    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"]
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.901933    5031 cpu_manager.go:414] "containerMap: removing container" podUID="357ec568-161a-41bb-bede-1eb9658d6461" containerName="main"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.901952    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357ec568-161a-41bb-bede-1eb9658d6461" containerName="main"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.901999    5031 cpu_manager.go:414] "containerMap: removing container" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902023    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902041    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902049    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902140    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c512156-80a9-4188-8cd7-937d9d23a4a5" containerName="istio-proxy"
Nov 01 23:30:53.902147 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902150    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fb8fcc-3571-4d82-bbc3-4f8b6756ef76" containerName="payload-processing"
Nov 01 23:30:53.902587 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.902164    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="357ec568-161a-41bb-bede-1eb9658d6461" containerName="main"
Nov 01 23:30:53.905490 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:53.905463    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.001321 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.001286    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"]
Nov 01 23:30:54.047413 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.047378    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d807183-2f45-40c4-9134-27450d27616a-openshift-service-ca\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.047672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.047429    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ffm\" (UniqueName: \"kubernetes.io/projected/1d807183-2f45-40c4-9134-27450d27616a-kube-api-access-s8ffm\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.047672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.047574    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-metrics-certs\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.047672 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.047637    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-maas-api-tls\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.149052 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.149011    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d807183-2f45-40c4-9134-27450d27616a-openshift-service-ca\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.149052 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.149056    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ffm\" (UniqueName: \"kubernetes.io/projected/1d807183-2f45-40c4-9134-27450d27616a-kube-api-access-s8ffm\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.149291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.149091    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-metrics-certs\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.149291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.149153    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-maas-api-tls\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.149719 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.149696    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d807183-2f45-40c4-9134-27450d27616a-openshift-service-ca\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.151954 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.151900    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-maas-api-tls\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.152059 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.151975    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d807183-2f45-40c4-9134-27450d27616a-metrics-certs\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.203871 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.203835    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ffm\" (UniqueName: \"kubernetes.io/projected/1d807183-2f45-40c4-9134-27450d27616a-kube-api-access-s8ffm\") pod \"maas-api-58d749f5db-b8lfc\" (UID: \"1d807183-2f45-40c4-9134-27450d27616a\") " pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.218708 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.218674    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:54.590053 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.590017    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"]
Nov 01 23:30:54.597094 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:30:54.597061    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d807183_2f45_40c4_9134_27450d27616a.slice/crio-9787295156c241176f46fc980fcb4be3fdaec0adabd583efd817e68d9a425d61 WatchSource:0}: Error finding container 9787295156c241176f46fc980fcb4be3fdaec0adabd583efd817e68d9a425d61: Status 404 returned error can't find the container with id 9787295156c241176f46fc980fcb4be3fdaec0adabd583efd817e68d9a425d61
Nov 01 23:30:54.598569 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:54.598533    5031 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:30:55.552252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:55.552219    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc" event={"ID":"1d807183-2f45-40c4-9134-27450d27616a","Type":"ContainerStarted","Data":"9787295156c241176f46fc980fcb4be3fdaec0adabd583efd817e68d9a425d61"}
Nov 01 23:30:56.558875 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:56.558829    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc" event={"ID":"1d807183-2f45-40c4-9134-27450d27616a","Type":"ContainerStarted","Data":"eae0f7370d41eeca9954f5fa0ce6a1b466e82038f55a0e489c2a7115a4d993b2"}
Nov 01 23:30:56.559295 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:56.558969    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:30:56.668524 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:30:56.668463    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc" podStartSLOduration=2.743144171 podStartE2EDuration="3.668448755s" totalImagesPullingTime="925.304584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:56.667730978 +0000 UTC m=+4010.623770823" watchObservedRunningTime="2026-11-01 23:30:56.668448755 +0000 UTC m=+4010.624488618"
Nov 01 23:31:00.713013 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.712979    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:31:00.716683 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.716657    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.766267 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.765592    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a38c0c87-openshift-default-dockercfg-mmdzr\""
Nov 01 23:31:00.812971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.812936    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.812971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.812972    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813083    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813101    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813224 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813189    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813355 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813271    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813355 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813318    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813425 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813375    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zdkp\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.813425 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.813404    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.821353 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.821309    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:31:00.913986 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.913939    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914200 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914034    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914200 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914114    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914200 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914138    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914376 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914290    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914376 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914326    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914376 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914347    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914573 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914399    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zdkp\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914573 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914405    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914573 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914457    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914573 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914469    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914758 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914665    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.914758 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.914721    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.915064 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.915040    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.916831 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.916808    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.916971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.916952    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.967600 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.967511    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:00.967780 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:00.967689    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zdkp\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\") pod \"e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") " pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:01.029829 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.029784    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:01.456273 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.456233    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:31:01.462657 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:31:01.462626    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198c5dbb_84df_4b18_9e5b_b7ce4ce8284f.slice/crio-dcf848a117f62988f59bf8a8ca511d0e47a5cab0d2be1e720219cb611fe733b4 WatchSource:0}: Error finding container dcf848a117f62988f59bf8a8ca511d0e47a5cab0d2be1e720219cb611fe733b4: Status 404 returned error can't find the container with id dcf848a117f62988f59bf8a8ca511d0e47a5cab0d2be1e720219cb611fe733b4
Nov 01 23:31:01.464886 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.464853    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:01.464978 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.464921    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:01.464978 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.464950    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:01.591246 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.583981    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerStarted","Data":"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"}
Nov 01 23:31:01.591246 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.584084    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerStarted","Data":"dcf848a117f62988f59bf8a8ca511d0e47a5cab0d2be1e720219cb611fe733b4"}
Nov 01 23:31:01.711966 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:01.711857    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podStartSLOduration=1.711841344 podStartE2EDuration="1.711841344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:31:01.711203317 +0000 UTC m=+4015.667243163" watchObservedRunningTime="2026-11-01 23:31:01.711841344 +0000 UTC m=+4015.667881187"
Nov 01 23:31:02.030968 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.030878    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:02.036433 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.036401    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:02.569649 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.569621    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d749f5db-b8lfc"
Nov 01 23:31:02.588344 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.588317    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:02.589300 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.589281    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:02.724888 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.724853    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 23:31:02.725132 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:02.725100    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" podUID="0adb6621-c7c7-42be-be85-804dc8a64f04" containerName="maas-api" containerID="cri-o://1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad" gracePeriod=30
Nov 01 23:31:04.084975 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.084944    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 23:31:04.142691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.142595    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\") pod \"0adb6621-c7c7-42be-be85-804dc8a64f04\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") "
Nov 01 23:31:04.142691 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.142649    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\" (UniqueName: \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\") pod \"0adb6621-c7c7-42be-be85-804dc8a64f04\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") "
Nov 01 23:31:04.142904 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.142697    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\") pod \"0adb6621-c7c7-42be-be85-804dc8a64f04\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") "
Nov 01 23:31:04.142904 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.142742    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\") pod \"0adb6621-c7c7-42be-be85-804dc8a64f04\" (UID: \"0adb6621-c7c7-42be-be85-804dc8a64f04\") "
Nov 01 23:31:04.143064 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.143039    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca" pod "0adb6621-c7c7-42be-be85-804dc8a64f04" (UID: "0adb6621-c7c7-42be-be85-804dc8a64f04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:04.145096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.145063    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs" pod "0adb6621-c7c7-42be-be85-804dc8a64f04" (UID: "0adb6621-c7c7-42be-be85-804dc8a64f04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:04.145096 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.145088    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls" pod "0adb6621-c7c7-42be-be85-804dc8a64f04" (UID: "0adb6621-c7c7-42be-be85-804dc8a64f04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:04.145312 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.145152    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22" pod "0adb6621-c7c7-42be-be85-804dc8a64f04" (UID: "0adb6621-c7c7-42be-be85-804dc8a64f04"). InnerVolumeSpecName "kube-api-access-kpm22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:04.244158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.244121    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adb6621-c7c7-42be-be85-804dc8a64f04-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:04.244158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.244153    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpm22\" (UniqueName: \"kubernetes.io/projected/0adb6621-c7c7-42be-be85-804dc8a64f04-kube-api-access-kpm22\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:04.244158 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.244164    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:04.244398 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.244173    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adb6621-c7c7-42be-be85-804dc8a64f04-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:04.600604 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.600563    5031 generic.go:356] "Generic (PLEG): container finished" podID="0adb6621-c7c7-42be-be85-804dc8a64f04" containerID="1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad" exitCode=0
Nov 01 23:31:04.600790 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.600681    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"
Nov 01 23:31:04.600790 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.600714    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" event={"ID":"0adb6621-c7c7-42be-be85-804dc8a64f04","Type":"ContainerDied","Data":"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"}
Nov 01 23:31:04.600790 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.600774    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64" event={"ID":"0adb6621-c7c7-42be-be85-804dc8a64f04","Type":"ContainerDied","Data":"f7cb9109800dc9003b9288b7c040a5c57062fd97c95e1343bbcf48b0a18e95b2"}
Nov 01 23:31:04.600967 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.600800    5031 scope.go:122] "RemoveContainer" containerID="1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"
Nov 01 23:31:04.609555 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.609515    5031 scope.go:122] "RemoveContainer" containerID="1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"
Nov 01 23:31:04.609936 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:04.609906    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad\": container with ID starting with 1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad not found: ID does not exist" containerID="1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"
Nov 01 23:31:04.610072 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.609950    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad"} err="failed to get container status \"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad\": rpc error: code = NotFound desc = could not find container \"1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad\": container with ID starting with 1d44379c383ba1a461518e567b01362d198308bd3951e7060214387cf9e383ad not found: ID does not exist"
Nov 01 23:31:04.754861 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.754827    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 23:31:04.801130 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:04.801092    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbcdfbbb-2nn64"]
Nov 01 23:31:06.169470 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:06.169423    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adb6621-c7c7-42be-be85-804dc8a64f04" path="/var/lib/kubelet/pods/0adb6621-c7c7-42be-be85-804dc8a64f04/volumes"
Nov 01 23:31:07.582726 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.582690    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:07.583169 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.583091    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0adb6621-c7c7-42be-be85-804dc8a64f04" containerName="maas-api"
Nov 01 23:31:07.583169 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.583102    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adb6621-c7c7-42be-be85-804dc8a64f04" containerName="maas-api"
Nov 01 23:31:07.583169 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.583161    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adb6621-c7c7-42be-be85-804dc8a64f04" containerName="maas-api"
Nov 01 23:31:07.587793 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.587767    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.636367 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.636331    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a38c0c87\""
Nov 01 23:31:07.675120 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.675075    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljtz\" (UniqueName: \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.675331 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.675149    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.682844 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.682807    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a38c0c87-dockercfg-q7f6f\""
Nov 01 23:31:07.685353 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.685316    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:07.730433 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.730393    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:07.734050 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.734023    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.776476 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.776431    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.776678 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.776485    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgb44\" (UniqueName: \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.776678 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.776523    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.776754 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.776699    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljtz\" (UniqueName: \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.777173 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.777152    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.786078 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.786049    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:07.830807 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.830774    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljtz\" (UniqueName: \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\") pod \"payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.877367 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.877272    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgb44\" (UniqueName: \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.877367 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.877320    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.878179 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.878158    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:07.900958 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.900925    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:07.933433 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:07.933392    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgb44\" (UniqueName: \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\") pod \"payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") " pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:08.046915 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.046880    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:08.276245 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.276138    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:08.283068 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:31:08.283034    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7fab4f_e228_4a8a_a75f_ff5e4ef9376b.slice/crio-1fc9a9447a1edb2aef8da74ae1bbee7e6df2b4ef6457b3056d8d621c52b1ddf2 WatchSource:0}: Error finding container 1fc9a9447a1edb2aef8da74ae1bbee7e6df2b4ef6457b3056d8d621c52b1ddf2: Status 404 returned error can't find the container with id 1fc9a9447a1edb2aef8da74ae1bbee7e6df2b4ef6457b3056d8d621c52b1ddf2
Nov 01 23:31:08.417862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.417826    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:08.424815 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:31:08.424784    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ee729_fb69_4a58_a401_448bfb035db6.slice/crio-b6fc532ca312c9071a0e19911a5bb86588dddb80062a6c3ac9682816dc747e34 WatchSource:0}: Error finding container b6fc532ca312c9071a0e19911a5bb86588dddb80062a6c3ac9682816dc747e34: Status 404 returned error can't find the container with id b6fc532ca312c9071a0e19911a5bb86588dddb80062a6c3ac9682816dc747e34
Nov 01 23:31:08.622229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.622192    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" event={"ID":"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b","Type":"ContainerStarted","Data":"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"}
Nov 01 23:31:08.622959 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.622266    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" event={"ID":"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b","Type":"ContainerStarted","Data":"1fc9a9447a1edb2aef8da74ae1bbee7e6df2b4ef6457b3056d8d621c52b1ddf2"}
Nov 01 23:31:08.622959 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.622392    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:08.624044 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.624014    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" event={"ID":"7e2ee729-fb69-4a58-a401-448bfb035db6","Type":"ContainerStarted","Data":"4e3a198e3d85454efa3f39a04d904a123bf8f98e17e2e7497d41285e21c49f58"}
Nov 01 23:31:08.624155 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.624064    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" event={"ID":"7e2ee729-fb69-4a58-a401-448bfb035db6","Type":"ContainerStarted","Data":"b6fc532ca312c9071a0e19911a5bb86588dddb80062a6c3ac9682816dc747e34"}
Nov 01 23:31:08.624155 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.624091    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:08.731586 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.731518    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" podStartSLOduration=1.731504556 podStartE2EDuration="1.731504556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:31:08.729762807 +0000 UTC m=+4022.685802652" watchObservedRunningTime="2026-11-01 23:31:08.731504556 +0000 UTC m=+4022.687544391"
Nov 01 23:31:08.836972 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:08.836909    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" podStartSLOduration=1.836891797 podStartE2EDuration="1.836891797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:31:08.835963608 +0000 UTC m=+4022.792003446" watchObservedRunningTime="2026-11-01 23:31:08.836891797 +0000 UTC m=+4022.792931635"
Nov 01 23:31:09.257370 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.257333    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:31:09.262340 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.262313    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.292229 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.292188    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9c6\" (UniqueName: \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.292390 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.292238    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.292434 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.292391    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.292434 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.292421    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.311211 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.311178    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a38c0c87-serving-cert\""
Nov 01 23:31:09.325917 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.325873    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:31:09.393341 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.393309    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9c6\" (UniqueName: \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.393563 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.393368    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.393563 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.393489    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.393563 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.393513    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.393563 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:09.393522    5031 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-serving-cert: secret "maas-api-e2e-ait-a38c0c87-serving-cert" not found
Nov 01 23:31:09.393801 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:09.393624    5031 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls podName:0f3b42a6-87f7-4a78-9e11-36e7f53d5c10 nodeName:}" failed. No retries permitted until 2026-11-01 23:31:09.893602014 +0000 UTC m=+4023.849641858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls") pod "maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" (UID: "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10") : secret "maas-api-e2e-ait-a38c0c87-serving-cert" not found
Nov 01 23:31:09.394406 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.394381    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.396289 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.396271    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.449929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.449895    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9c6\" (UniqueName: \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.899608 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.899533    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:09.902360 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:09.902333    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") pod \"maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:10.175862 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:10.175767    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:10.544847 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:10.544812    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:31:10.551653 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:31:10.551613    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3b42a6_87f7_4a78_9e11_36e7f53d5c10.slice/crio-f900329fe55a6a2bf9a8ad21bc9127522670d77a14e3468ce73a5f0366b9884f WatchSource:0}: Error finding container f900329fe55a6a2bf9a8ad21bc9127522670d77a14e3468ce73a5f0366b9884f: Status 404 returned error can't find the container with id f900329fe55a6a2bf9a8ad21bc9127522670d77a14e3468ce73a5f0366b9884f
Nov 01 23:31:10.633449 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:10.633405    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:31:10.636840 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:10.636807    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" event={"ID":"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10","Type":"ContainerStarted","Data":"f900329fe55a6a2bf9a8ad21bc9127522670d77a14e3468ce73a5f0366b9884f"}
Nov 01 23:31:11.642726 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:11.642689    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" event={"ID":"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10","Type":"ContainerStarted","Data":"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"}
Nov 01 23:31:11.643123 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:11.642774    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:11.753234 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:11.753120    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" podStartSLOduration=1.826820597 podStartE2EDuration="2.753098376s" totalImagesPullingTime="926.277779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:31:11.751715065 +0000 UTC m=+4025.707754907" watchObservedRunningTime="2026-11-01 23:31:11.753098376 +0000 UTC m=+4025.709138254"
Nov 01 23:31:17.668564 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:17.657852    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:19.632004 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:19.631975    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:19.632413 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:19.632062    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:26.722858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.722774    5031 generic.go:356] "Generic (PLEG): container finished" podID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerID="8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f" exitCode=0
Nov 01 23:31:26.722858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.722828    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerDied","Data":"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"}
Nov 01 23:31:26.723323 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.723242    5031 scope.go:122] "RemoveContainer" containerID="8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"
Nov 01 23:31:26.724684 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.724643    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:26.724757 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.724732    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:26.724800 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:26.724774    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:31:27.731047 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:27.731013    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerStarted","Data":"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"}
Nov 01 23:31:28.030526 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:28.030439    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:28.031881 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:28.031856    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 23:31:28.031946 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:28.031909    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 23:31:29.030971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:29.030935    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 23:31:29.031410 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:29.030988    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 23:31:30.030828 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:30.030789    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 23:31:30.030996 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:30.030851    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 23:31:31.030511 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:31.030474    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 23:31:31.030749 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:31.030528    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 23:31:32.030697 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:32.030661    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 01 23:31:32.031117 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:32.030721    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 01 23:31:32.588647 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:32.588612    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:33.034727 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:33.034627    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:33.035732 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:33.035701    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:31:40.625154 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:40.625107    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:31:51.715874 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:51.715781    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:51.823666 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:51.823624    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:56.716533 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:56.716469    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" podUID="7e2ee729-fb69-4a58-a401-448bfb035db6" containerName="payload-processing" containerID="cri-o://4e3a198e3d85454efa3f39a04d904a123bf8f98e17e2e7497d41285e21c49f58" gracePeriod=25
Nov 01 23:31:56.824559 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:56.824457    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" podUID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" containerName="payload-pre-processing" containerID="cri-o://30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce" gracePeriod=25
Nov 01 23:31:56.876182 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:56.876140    5031 generic.go:356] "Generic (PLEG): container finished" podID="7e2ee729-fb69-4a58-a401-448bfb035db6" containerID="4e3a198e3d85454efa3f39a04d904a123bf8f98e17e2e7497d41285e21c49f58" exitCode=0
Nov 01 23:31:56.876357 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:56.876217    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" event={"ID":"7e2ee729-fb69-4a58-a401-448bfb035db6","Type":"ContainerDied","Data":"4e3a198e3d85454efa3f39a04d904a123bf8f98e17e2e7497d41285e21c49f58"}
Nov 01 23:31:56.976143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:56.975581    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:57.080820 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.080788    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:57.081182 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.081164    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\") pod \"7e2ee729-fb69-4a58-a401-448bfb035db6\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") "
Nov 01 23:31:57.081277 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.081224    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\" (UniqueName: \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\") pod \"7e2ee729-fb69-4a58-a401-448bfb035db6\" (UID: \"7e2ee729-fb69-4a58-a401-448bfb035db6\") "
Nov 01 23:31:57.081566 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.081520    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume" pod "7e2ee729-fb69-4a58-a401-448bfb035db6" (UID: "7e2ee729-fb69-4a58-a401-448bfb035db6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:57.081683 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.081650    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2ee729-fb69-4a58-a401-448bfb035db6-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:57.083416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.083396    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44" pod "7e2ee729-fb69-4a58-a401-448bfb035db6" (UID: "7e2ee729-fb69-4a58-a401-448bfb035db6"). InnerVolumeSpecName "kube-api-access-vgb44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:57.181984 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.181944    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\" (UniqueName: \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\") pod \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") "
Nov 01 23:31:57.182308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.182062    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\") pod \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\" (UID: \"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b\") "
Nov 01 23:31:57.182385 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.182359    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgb44\" (UniqueName: \"kubernetes.io/projected/7e2ee729-fb69-4a58-a401-448bfb035db6-kube-api-access-vgb44\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:57.182435 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.182398    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume" pod "2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" (UID: "2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:57.184293 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.184270    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz" pod "2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" (UID: "2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b"). InnerVolumeSpecName "kube-api-access-dljtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:57.283220 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.283116    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:57.283220 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.283150    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljtz\" (UniqueName: \"kubernetes.io/projected/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b-kube-api-access-dljtz\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:57.882658 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.882626    5031 generic.go:356] "Generic (PLEG): container finished" podID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" containerID="30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce" exitCode=0
Nov 01 23:31:57.883149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.882676    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"
Nov 01 23:31:57.883149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.882702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" event={"ID":"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b","Type":"ContainerDied","Data":"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"}
Nov 01 23:31:57.883149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.882754    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl" event={"ID":"2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b","Type":"ContainerDied","Data":"1fc9a9447a1edb2aef8da74ae1bbee7e6df2b4ef6457b3056d8d621c52b1ddf2"}
Nov 01 23:31:57.883149 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.882785    5031 scope.go:122] "RemoveContainer" containerID="30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"
Nov 01 23:31:57.884193 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.884168    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj" event={"ID":"7e2ee729-fb69-4a58-a401-448bfb035db6","Type":"ContainerDied","Data":"b6fc532ca312c9071a0e19911a5bb86588dddb80062a6c3ac9682816dc747e34"}
Nov 01 23:31:57.884269 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.884218    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"
Nov 01 23:31:57.892903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.892876    5031 scope.go:122] "RemoveContainer" containerID="30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"
Nov 01 23:31:57.893245 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:57.893224    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce\": container with ID starting with 30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce not found: ID does not exist" containerID="30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"
Nov 01 23:31:57.893303 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.893254    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce"} err="failed to get container status \"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce\": rpc error: code = NotFound desc = could not find container \"30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce\": container with ID starting with 30660a811972f757b01801240b3b0c7710dbd1bbf4b717079a8a576c6f01e8ce not found: ID does not exist"
Nov 01 23:31:57.893303 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:57.893272    5031 scope.go:122] "RemoveContainer" containerID="4e3a198e3d85454efa3f39a04d904a123bf8f98e17e2e7497d41285e21c49f58"
Nov 01 23:31:58.035241 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.035159    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:58.081495 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.081456    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a38c0c87-7fd9bbc9cf-clzjl"]
Nov 01 23:31:58.142302 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.142265    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:58.146481 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.146440    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a38c0c87-7cf9777fcc-f6fhj"]
Nov 01 23:31:58.170699 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.170670    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" path="/var/lib/kubelet/pods/2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b/volumes"
Nov 01 23:31:58.171073 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.171059    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ee729-fb69-4a58-a401-448bfb035db6" path="/var/lib/kubelet/pods/7e2ee729-fb69-4a58-a401-448bfb035db6/volumes"
Nov 01 23:31:58.213894 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.213858    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:31:58.214128 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.214102    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" podUID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" containerName="maas-api" containerID="cri-o://f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd" gracePeriod=30
Nov 01 23:31:58.470624 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.470597    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:58.597442 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.597353    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\" (UniqueName: \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\") pod \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") "
Nov 01 23:31:58.597442 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.597398    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") pod \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") "
Nov 01 23:31:58.597731 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.597452    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\") pod \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") "
Nov 01 23:31:58.597731 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.597484    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\") pod \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\" (UID: \"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10\") "
Nov 01 23:31:58.598577 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.598519    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca" pod "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" (UID: "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:58.599969 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.599940    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs" pod "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" (UID: "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:58.600092 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.600014    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6" pod "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" (UID: "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10"). InnerVolumeSpecName "kube-api-access-wx9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:58.600294 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.600265    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls" pod "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" (UID: "0f3b42a6-87f7-4a78-9e11-36e7f53d5c10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:58.698982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.698939    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx9c6\" (UniqueName: \"kubernetes.io/projected/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-kube-api-access-wx9c6\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:58.698982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.698977    5031 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-maas-api-tls\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:58.698982 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.698988    5031 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-openshift-service-ca\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:58.699237 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.698997    5031 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10-metrics-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:58.892646 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.892520    5031 generic.go:356] "Generic (PLEG): container finished" podID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" containerID="f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd" exitCode=0
Nov 01 23:31:58.892646 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.892584    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"
Nov 01 23:31:58.893210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.892588    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" event={"ID":"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10","Type":"ContainerDied","Data":"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"}
Nov 01 23:31:58.893210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.892702    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9" event={"ID":"0f3b42a6-87f7-4a78-9e11-36e7f53d5c10","Type":"ContainerDied","Data":"f900329fe55a6a2bf9a8ad21bc9127522670d77a14e3468ce73a5f0366b9884f"}
Nov 01 23:31:58.893210 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.892724    5031 scope.go:122] "RemoveContainer" containerID="f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"
Nov 01 23:31:58.902656 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.902630    5031 scope.go:122] "RemoveContainer" containerID="f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"
Nov 01 23:31:58.902979 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:31:58.902957    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd\": container with ID starting with f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd not found: ID does not exist" containerID="f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"
Nov 01 23:31:58.903034 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:58.902989    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd"} err="failed to get container status \"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd\": rpc error: code = NotFound desc = could not find container \"f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd\": container with ID starting with f9664c778c7ee2e03039667efd7fe6b28a4f16b5016a53752b53de0d80b070cd not found: ID does not exist"
Nov 01 23:31:59.045347 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:59.045311    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:31:59.091437 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:31:59.091399    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a38c0c87-55f7787cd4-69wr9"]
Nov 01 23:32:00.170278 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:00.170242    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" path="/var/lib/kubelet/pods/0f3b42a6-87f7-4a78-9e11-36e7f53d5c10/volumes"
Nov 01 23:32:01.187258 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:01.187222    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:32:01.187736 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:01.187470    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" containerID="cri-o://f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6" gracePeriod=30
Nov 01 23:32:03.037869 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.037833    5031 patch_prober.go:29] interesting pod/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 01 23:32:03.038247 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.037894    5031 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:32:03.170849 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.170816    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:32:03.171206 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171194    5031 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ee729-fb69-4a58-a401-448bfb035db6" containerName="payload-processing"
Nov 01 23:32:03.171254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171207    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ee729-fb69-4a58-a401-448bfb035db6" containerName="payload-processing"
Nov 01 23:32:03.171254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171228    5031 cpu_manager.go:414] "containerMap: removing container" podUID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" containerName="payload-pre-processing"
Nov 01 23:32:03.171254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171234    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" containerName="payload-pre-processing"
Nov 01 23:32:03.171254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171242    5031 cpu_manager.go:414] "containerMap: removing container" podUID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" containerName="maas-api"
Nov 01 23:32:03.171254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171247    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" containerName="maas-api"
Nov 01 23:32:03.171408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171301    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7fab4f-e228-4a8a-a75f-ff5e4ef9376b" containerName="payload-pre-processing"
Nov 01 23:32:03.171408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171310    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3b42a6-87f7-4a78-9e11-36e7f53d5c10" containerName="maas-api"
Nov 01 23:32:03.171408 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.171320    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ee729-fb69-4a58-a401-448bfb035db6" containerName="payload-processing"
Nov 01 23:32:03.175798 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.175774    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.226317 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.226270    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-2474f7-openshift-default-dockercfg-756w5\""
Nov 01 23:32:03.238939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.238891    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.238939 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.238944    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.238968    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239090    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239235    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239316 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239288    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239320    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8snc\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239490 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239476    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.239588 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.239571    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.280631 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.280590    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:32:03.340127 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340090    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340137    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340154    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8snc\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340204    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340315 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340251    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340398    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340461    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340501    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340599 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340528    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340774 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340637    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340774 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340732    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340881    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.340961 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.340915    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.341164 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.341106    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.342761 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.342739    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.343093 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.343074    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.412004 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.411962    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8snc\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.412420 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.412395    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\") pod \"e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") " pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.489139 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.489106    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:03.918778 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.918740    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:32:03.925170 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:32:03.925139    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e81b60c_148a_431a_b336_e0601f0bbf4d.slice/crio-bd7cd4ad5a8a8de09cdf3490a66ee488a4746568f5fb0c0320a3c8e6d7359b14 WatchSource:0}: Error finding container bd7cd4ad5a8a8de09cdf3490a66ee488a4746568f5fb0c0320a3c8e6d7359b14: Status 404 returned error can't find the container with id bd7cd4ad5a8a8de09cdf3490a66ee488a4746568f5fb0c0320a3c8e6d7359b14
Nov 01 23:32:03.927409 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.927372    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:03.927515 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.927443    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:03.927515 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:03.927473    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:04.923225 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:04.923189    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerStarted","Data":"6389e75d0cfd13e8519ca6756126d000d759e249c2589a8272b599f567d25448"}
Nov 01 23:32:04.923788 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:04.923261    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerStarted","Data":"bd7cd4ad5a8a8de09cdf3490a66ee488a4746568f5fb0c0320a3c8e6d7359b14"}
Nov 01 23:32:05.129771 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:05.129709    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podStartSLOduration=2.129692353 podStartE2EDuration="2.129692353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:32:05.082229251 +0000 UTC m=+4079.038269095" watchObservedRunningTime="2026-11-01 23:32:05.129692353 +0000 UTC m=+4079.085732196"
Nov 01 23:32:05.490205 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:05.490169    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:05.495723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:05.495686    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:05.928351 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:05.928317    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:05.929292 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:05.929275    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:06.431018 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.430988    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:32:06.473407 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473318    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473407 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473372    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473407 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473398    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473429    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473489    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473527    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473640    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473723 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473720    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473751    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\") pod \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\" (UID: \"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f\") "
Nov 01 23:32:06.473991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473821    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:32:06.473991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473853    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:32:06.473991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.473881    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474025    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474108    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474153    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474172    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474181    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.474195 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.474189    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.475971 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.475948    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:32:06.476074 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.475965    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:32:06.476426 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.476408    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "kube-api-access-7zdkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:32:06.476487 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.476424    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token" pod "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" (UID: "198c5dbb-84df-4b18-9e5b-b7ce4ce8284f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:32:06.574991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.574946    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.574991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.574987    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zdkp\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-kube-api-access-7zdkp\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.574991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.574999    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.575232 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.575008    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.575232 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.575017    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:32:06.934786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.934748    5031 generic.go:356] "Generic (PLEG): container finished" podID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerID="f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6" exitCode=0
Nov 01 23:32:06.935187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.934810    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"
Nov 01 23:32:06.935187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.934830    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerDied","Data":"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"}
Nov 01 23:32:06.935187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.934895    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz" event={"ID":"198c5dbb-84df-4b18-9e5b-b7ce4ce8284f","Type":"ContainerDied","Data":"dcf848a117f62988f59bf8a8ca511d0e47a5cab0d2be1e720219cb611fe733b4"}
Nov 01 23:32:06.935187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.934919    5031 scope.go:122] "RemoveContainer" containerID="f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"
Nov 01 23:32:06.944131 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.944109    5031 scope.go:122] "RemoveContainer" containerID="8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"
Nov 01 23:32:06.952639 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.952610    5031 scope.go:122] "RemoveContainer" containerID="f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"
Nov 01 23:32:06.952977 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:32:06.952957    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6\": container with ID starting with f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6 not found: ID does not exist" containerID="f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"
Nov 01 23:32:06.953037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.952989    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6"} err="failed to get container status \"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6\": rpc error: code = NotFound desc = could not find container \"f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6\": container with ID starting with f0fa08588b8e80042028194e60a52adf6b20d65d3d80487e03bb8d66308c1db6 not found: ID does not exist"
Nov 01 23:32:06.953037 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.953010    5031 scope.go:122] "RemoveContainer" containerID="8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"
Nov 01 23:32:06.953263 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:32:06.953246    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f\": container with ID starting with 8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f not found: ID does not exist" containerID="8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"
Nov 01 23:32:06.953329 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:06.953265    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f"} err="failed to get container status \"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f\": rpc error: code = NotFound desc = could not find container \"8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f\": container with ID starting with 8ca93971092dca124da9e784e6710f3e8a5f9b1040e87f6c3f786b196960e37f not found: ID does not exist"
Nov 01 23:32:07.088610 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:07.088577    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:32:07.134844 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:07.134804    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a38c0c87-openshift-default-5c8f44f85f-zxdfz"]
Nov 01 23:32:08.170779 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:08.170743    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" path="/var/lib/kubelet/pods/198c5dbb-84df-4b18-9e5b-b7ce4ce8284f/volumes"
Nov 01 23:32:10.626648 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:32:10.626601    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:32:11.443819 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.443785    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:32:11.444254 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444235    5031 cpu_manager.go:414] "containerMap: removing container" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.444345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444256    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.444345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444272    5031 cpu_manager.go:414] "containerMap: removing container" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.444345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444279    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.444499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444395    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.444499 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.444407    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c5dbb-84df-4b18-9e5b-b7ce4ce8284f" containerName="istio-proxy"
Nov 01 23:32:11.447483 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.447453    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.522685 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.522646    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xxl\" (UniqueName: \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.522878 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.522833    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.553234 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.553192    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-2474f7-dockercfg-bf678\""
Nov 01 23:32:11.554296 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.554268    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-2474f7\""
Nov 01 23:32:11.556624 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.555487    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:32:11.600612 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.600570    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:32:11.604184 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.604165    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.624299 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.624243    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.624477 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.624323    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xxl\" (UniqueName: \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.625075 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.625053    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.659718 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.659681    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:32:11.679991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.679949    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xxl\" (UniqueName: \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\") pod \"payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.725589 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.725463    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxl9\" (UniqueName: \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.725589 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.725512    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.760440 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.760396    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:11.827385 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.827347    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxl9\" (UniqueName: \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.827385 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.827389    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.828143 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.828117    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.882412 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.882376    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxl9\" (UniqueName: \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\") pod \"payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:11.917339 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:11.917302    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:12.128418 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.128384    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:32:12.135067 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:32:12.135038    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508f8a14_faac_452a_9b5c_1a31fe14ed3c.slice/crio-8ad37d64cb68eba0cdcde68d22904303b76c2979619d001a0fdb94cef4df0674 WatchSource:0}: Error finding container 8ad37d64cb68eba0cdcde68d22904303b76c2979619d001a0fdb94cef4df0674: Status 404 returned error can't find the container with id 8ad37d64cb68eba0cdcde68d22904303b76c2979619d001a0fdb94cef4df0674
Nov 01 23:32:12.290263 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.290180    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:32:12.296381 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:32:12.296339    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a39cb3_50e2_42ff_8465_af0257488786.slice/crio-b27bafa3dd951dd7bc0b0b992ad2d24cda456d0de6cbaec862df7f7133d39441 WatchSource:0}: Error finding container b27bafa3dd951dd7bc0b0b992ad2d24cda456d0de6cbaec862df7f7133d39441: Status 404 returned error can't find the container with id b27bafa3dd951dd7bc0b0b992ad2d24cda456d0de6cbaec862df7f7133d39441
Nov 01 23:32:12.966556 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.966505    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" event={"ID":"f9a39cb3-50e2-42ff-8465-af0257488786","Type":"ContainerStarted","Data":"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"}
Nov 01 23:32:12.967002 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.966594    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:12.967002 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.966620    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" event={"ID":"f9a39cb3-50e2-42ff-8465-af0257488786","Type":"ContainerStarted","Data":"b27bafa3dd951dd7bc0b0b992ad2d24cda456d0de6cbaec862df7f7133d39441"}
Nov 01 23:32:12.968369 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.968331    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" event={"ID":"508f8a14-faac-452a-9b5c-1a31fe14ed3c","Type":"ContainerStarted","Data":"625050a4ba9c3af6c2b2fcebea2e0c7054c65a495c800492ef219c709f7dc78f"}
Nov 01 23:32:12.968522 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.968388    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" event={"ID":"508f8a14-faac-452a-9b5c-1a31fe14ed3c","Type":"ContainerStarted","Data":"8ad37d64cb68eba0cdcde68d22904303b76c2979619d001a0fdb94cef4df0674"}
Nov 01 23:32:12.968522 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:12.968411    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:13.075771 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:13.075714    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" podStartSLOduration=2.075698932 podStartE2EDuration="2.075698932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:32:13.07399836 +0000 UTC m=+4087.030038204" watchObservedRunningTime="2026-11-01 23:32:13.075698932 +0000 UTC m=+4087.031738770"
Nov 01 23:32:13.182068 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:13.182004    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" podStartSLOduration=2.181988283 podStartE2EDuration="2.181988283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:32:13.181134672 +0000 UTC m=+4087.137174517" watchObservedRunningTime="2026-11-01 23:32:13.181988283 +0000 UTC m=+4087.138028127"
Nov 01 23:32:23.980968 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:23.980923    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:32:23.981480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:23.981224    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:32:33.066416 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.066381    5031 generic.go:356] "Generic (PLEG): container finished" podID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerID="6389e75d0cfd13e8519ca6756126d000d759e249c2589a8272b599f567d25448" exitCode=0
Nov 01 23:32:33.066914 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.066452    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerDied","Data":"6389e75d0cfd13e8519ca6756126d000d759e249c2589a8272b599f567d25448"}
Nov 01 23:32:33.066914 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.066826    5031 scope.go:122] "RemoveContainer" containerID="6389e75d0cfd13e8519ca6756126d000d759e249c2589a8272b599f567d25448"
Nov 01 23:32:33.068047 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.068018    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:33.068128 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.068092    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:33.068165 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:33.068126    5031 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 01 23:32:34.084142 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:34.084105    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerStarted","Data":"235ac24990e87a34cf10f56fd44c1edcb6015ebc359203d5359213b58842d081"}
Nov 01 23:32:34.490383 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:34.490295    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:34.491700 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:34.491677    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 23:32:34.491808 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:34.491725    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 23:32:35.490501 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:35.490466    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 23:32:35.490943 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:35.490522    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 23:32:35.929215 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:35.929177    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:36.490243 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:36.490203    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 23:32:36.490418 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:36.490265    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 23:32:37.489674 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:37.489638    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 23:32:37.490144 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:37.489710    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 23:32:38.489921 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:38.489876    5031 patch_prober.go:29] interesting pod/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 01 23:32:38.490328 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:38.489951    5031 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 01 23:32:39.493457 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:39.493427    5031 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:39.494997 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:32:39.494978    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:32:40.625779 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:32:40.625739    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:33:10.625659 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:33:10.625617    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:33:40.626018 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:33:40.625936    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:34:07.260065 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:07.260024    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:34:07.367808 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:07.367772    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:34:10.625458 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:34:10.625415    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:34:12.261115 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.261045    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" podUID="f9a39cb3-50e2-42ff-8465-af0257488786" containerName="payload-processing" containerID="cri-o://bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b" gracePeriod=25
Nov 01 23:34:12.368872 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.368775    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" podUID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" containerName="payload-pre-processing" containerID="cri-o://625050a4ba9c3af6c2b2fcebea2e0c7054c65a495c800492ef219c709f7dc78f" gracePeriod=25
Nov 01 23:34:12.520199 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.520116    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:34:12.555640 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.555608    5031 generic.go:356] "Generic (PLEG): container finished" podID="f9a39cb3-50e2-42ff-8465-af0257488786" containerID="bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b" exitCode=0
Nov 01 23:34:12.555858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.555690    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"
Nov 01 23:34:12.555858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.555690    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" event={"ID":"f9a39cb3-50e2-42ff-8465-af0257488786","Type":"ContainerDied","Data":"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"}
Nov 01 23:34:12.555858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.555755    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s" event={"ID":"f9a39cb3-50e2-42ff-8465-af0257488786","Type":"ContainerDied","Data":"b27bafa3dd951dd7bc0b0b992ad2d24cda456d0de6cbaec862df7f7133d39441"}
Nov 01 23:34:12.555858 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.555787    5031 scope.go:122] "RemoveContainer" containerID="bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"
Nov 01 23:34:12.557950 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.557917    5031 generic.go:356] "Generic (PLEG): container finished" podID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" containerID="625050a4ba9c3af6c2b2fcebea2e0c7054c65a495c800492ef219c709f7dc78f" exitCode=0
Nov 01 23:34:12.558122 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.557992    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" event={"ID":"508f8a14-faac-452a-9b5c-1a31fe14ed3c","Type":"ContainerDied","Data":"625050a4ba9c3af6c2b2fcebea2e0c7054c65a495c800492ef219c709f7dc78f"}
Nov 01 23:34:12.565568 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.565520    5031 scope.go:122] "RemoveContainer" containerID="bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"
Nov 01 23:34:12.565930 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:34:12.565900    5031 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b\": container with ID starting with bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b not found: ID does not exist" containerID="bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"
Nov 01 23:34:12.566055 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.565945    5031 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b"} err="failed to get container status \"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b\": rpc error: code = NotFound desc = could not find container \"bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b\": container with ID starting with bee9843a5c56e8463745b8c3a8c9dcb98ad3dcfd41aa2d114cf9292f3800603b not found: ID does not exist"
Nov 01 23:34:12.626265 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.626235    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:34:12.662686 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.662643    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\") pod \"f9a39cb3-50e2-42ff-8465-af0257488786\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") "
Nov 01 23:34:12.662896 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.662717    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\" (UniqueName: \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\") pod \"f9a39cb3-50e2-42ff-8465-af0257488786\" (UID: \"f9a39cb3-50e2-42ff-8465-af0257488786\") "
Nov 01 23:34:12.663040 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.663017    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume" pod "f9a39cb3-50e2-42ff-8465-af0257488786" (UID: "f9a39cb3-50e2-42ff-8465-af0257488786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:34:12.664929 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.664902    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9" pod "f9a39cb3-50e2-42ff-8465-af0257488786" (UID: "f9a39cb3-50e2-42ff-8465-af0257488786"). InnerVolumeSpecName "kube-api-access-fwxl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:12.764105 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.764063    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\" (UniqueName: \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\") pod \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") "
Nov 01 23:34:12.764303 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.764191    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\") pod \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\" (UID: \"508f8a14-faac-452a-9b5c-1a31fe14ed3c\") "
Nov 01 23:34:12.764487 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.764470    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a39cb3-50e2-42ff-8465-af0257488786-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:12.764585 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.764492    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxl9\" (UniqueName: \"kubernetes.io/projected/f9a39cb3-50e2-42ff-8465-af0257488786-kube-api-access-fwxl9\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:12.764585 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.764516    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume" pod "508f8a14-faac-452a-9b5c-1a31fe14ed3c" (UID: "508f8a14-faac-452a-9b5c-1a31fe14ed3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:34:12.766312 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.766288    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl" pod "508f8a14-faac-452a-9b5c-1a31fe14ed3c" (UID: "508f8a14-faac-452a-9b5c-1a31fe14ed3c"). InnerVolumeSpecName "kube-api-access-55xxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:12.865481 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.865449    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xxl\" (UniqueName: \"kubernetes.io/projected/508f8a14-faac-452a-9b5c-1a31fe14ed3c-kube-api-access-55xxl\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:12.865638 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.865481    5031 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/508f8a14-faac-452a-9b5c-1a31fe14ed3c-plugins-config-volume\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:12.968671 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.968636    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:34:12.972361 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:12.972330    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-2474f7-6f4f666667-8nj9s"]
Nov 01 23:34:13.563783 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:13.563749    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"
Nov 01 23:34:13.564291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:13.563782    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj" event={"ID":"508f8a14-faac-452a-9b5c-1a31fe14ed3c","Type":"ContainerDied","Data":"8ad37d64cb68eba0cdcde68d22904303b76c2979619d001a0fdb94cef4df0674"}
Nov 01 23:34:13.564291 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:13.563838    5031 scope.go:122] "RemoveContainer" containerID="625050a4ba9c3af6c2b2fcebea2e0c7054c65a495c800492ef219c709f7dc78f"
Nov 01 23:34:13.729006 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:13.728970    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:34:13.773930 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:13.773898    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-2474f7-867f5dfb59-jpwgj"]
Nov 01 23:34:14.171226 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:14.171185    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" path="/var/lib/kubelet/pods/508f8a14-faac-452a-9b5c-1a31fe14ed3c/volumes"
Nov 01 23:34:14.171594 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:14.171581    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a39cb3-50e2-42ff-8465-af0257488786" path="/var/lib/kubelet/pods/f9a39cb3-50e2-42ff-8465-af0257488786/volumes"
Nov 01 23:34:25.307554 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:25.307498    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:34:25.308061 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:25.307942    5031 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy" containerID="cri-o://235ac24990e87a34cf10f56fd44c1edcb6015ebc359203d5359213b58842d081" gracePeriod=30
Nov 01 23:34:30.649903 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.649867    5031 generic.go:356] "Generic (PLEG): container finished" podID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerID="235ac24990e87a34cf10f56fd44c1edcb6015ebc359203d5359213b58842d081" exitCode=0
Nov 01 23:34:30.650405 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.649929    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerDied","Data":"235ac24990e87a34cf10f56fd44c1edcb6015ebc359203d5359213b58842d081"}
Nov 01 23:34:30.650405 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.649974    5031 scope.go:122] "RemoveContainer" containerID="6389e75d0cfd13e8519ca6756126d000d759e249c2589a8272b599f567d25448"
Nov 01 23:34:30.833049 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.833006    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:34:30.853024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.852937    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853024 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853010    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853032    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853053    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853082    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853122    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853187    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.853252 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853225    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853276    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853292    5031 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\") pod \"1e81b60c-148a-431a-b336-e0601f0bbf4d\" (UID: \"1e81b60c-148a-431a-b336-e0601f0bbf4d\") "
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853464    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853623    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853670    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853779    5031 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853801    5031 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-workload-certs\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853815    5031 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e81b60c-148a-431a-b336-e0601f0bbf4d-istiod-ca-cert\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853830    5031 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-credential-socket\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.866637 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.853835    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:34:30.891209 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.891146    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:34:30.894114 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.894025    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:30.898405 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.898345    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "kube-api-access-h8snc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:30.913093 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.913025    5031 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo" pod "1e81b60c-148a-431a-b336-e0601f0bbf4d" (UID: "1e81b60c-148a-431a-b336-e0601f0bbf4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:34:30.958345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.958196    5031 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-envoy\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.958345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.958240    5031 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8snc\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-kube-api-access-h8snc\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.958345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.958256    5031 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-token\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.958345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.958269    5031 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-data\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:30.958345 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:30.958286    5031 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e81b60c-148a-431a-b336-e0601f0bbf4d-istio-podinfo\") on node \"ip-10-0-138-155.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:31.658419 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:31.658310    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv" event={"ID":"1e81b60c-148a-431a-b336-e0601f0bbf4d","Type":"ContainerDied","Data":"bd7cd4ad5a8a8de09cdf3490a66ee488a4746568f5fb0c0320a3c8e6d7359b14"}
Nov 01 23:34:31.658419 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:31.658408    5031 scope.go:122] "RemoveContainer" containerID="235ac24990e87a34cf10f56fd44c1edcb6015ebc359203d5359213b58842d081"
Nov 01 23:34:31.659019 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:31.658584    5031 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"
Nov 01 23:34:31.860713 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:31.860678    5031 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:34:31.863831 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:31.863798    5031 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-2474f7-openshift-default-86997867fc-t8lxv"]
Nov 01 23:34:32.169681 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:34:32.169648    5031 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" path="/var/lib/kubelet/pods/1e81b60c-148a-431a-b336-e0601f0bbf4d/volumes"
Nov 01 23:34:40.624987 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:34:40.624942    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:35:10.651942 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:35:10.651903    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:35:40.624920 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:35:40.624880    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:35:48.406308 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:48.406281    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9lxd2_cfeef97a-4b32-4466-9af5-7f1977e4fa8b/server/0.log"
Nov 01 23:35:49.816235 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:49.816203    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d749f5db-b8lfc_1d807183-2f45-40c4-9134-27450d27616a/maas-api/0.log"
Nov 01 23:35:53.062138 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:53.062105    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-krsjt_f599a277-3519-4cbc-b948-7d6ada3183dc/kuadrant-console-plugin/0.log"
Nov 01 23:35:55.276794 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:55.276761    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-xwnhc_501fa75d-cc43-478c-8d1c-d85b342bf25e/kube-auth-proxy/0.log"
Nov 01 23:35:55.487968 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:55.487934    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-2zqr6_bc9e3876-1e29-4e71-af35-638422d47e53/istio-proxy/0.log"
Nov 01 23:35:55.959480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:55.959454    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-5zmng_8e04541d-5090-4ab5-b89a-1d04d3c0a148/istio-proxy/0.log"
Nov 01 23:35:56.219390 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:56.219297    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fx6mf_e35dfd80-6c4d-47df-9581-698a23fe2ed3/istio-proxy/0.log"
Nov 01 23:35:59.486253 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:59.486222    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vjfhb_d3a603f6-220c-44c4-aae2-13cd972b0187/main/0.log"
Nov 01 23:35:59.705752 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:35:59.705722    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vws56_80903051-820f-4d46-ac36-c364a4f19caf/main/0.log"
Nov 01 23:36:00.183914 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:00.183882    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c_6344d70b-d83b-41e7-b1ef-51648a96835a/main/0.log"
Nov 01 23:36:00.905313 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:00.905281    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx_0d80334c-9fa2-413b-94ea-a4c353344f29/main/0.log"
Nov 01 23:36:10.626114 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:36:10.626076    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:36:11.940186 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:11.940157    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9lxd2_cfeef97a-4b32-4466-9af5-7f1977e4fa8b/server/0.log"
Nov 01 23:36:13.164664 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:13.164614    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d749f5db-b8lfc_1d807183-2f45-40c4-9134-27450d27616a/maas-api/0.log"
Nov 01 23:36:16.074167 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:16.074097    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-krsjt_f599a277-3519-4cbc-b948-7d6ada3183dc/kuadrant-console-plugin/0.log"
Nov 01 23:36:18.062511 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:18.062473    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-xwnhc_501fa75d-cc43-478c-8d1c-d85b342bf25e/kube-auth-proxy/0.log"
Nov 01 23:36:18.277706 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:18.277670    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-2zqr6_bc9e3876-1e29-4e71-af35-638422d47e53/istio-proxy/0.log"
Nov 01 23:36:18.746480 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:18.746450    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-5zmng_8e04541d-5090-4ab5-b89a-1d04d3c0a148/istio-proxy/0.log"
Nov 01 23:36:19.003742 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:19.003667    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fx6mf_e35dfd80-6c4d-47df-9581-698a23fe2ed3/istio-proxy/0.log"
Nov 01 23:36:22.022402 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:22.022375    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-vjfhb_d3a603f6-220c-44c4-aae2-13cd972b0187/main/0.log"
Nov 01 23:36:22.271948 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:22.271914    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vws56_80903051-820f-4d46-ac36-c364a4f19caf/main/0.log"
Nov 01 23:36:22.746677 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:22.746650    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8c952c_6344d70b-d83b-41e7-b1ef-51648a96835a/main/0.log"
Nov 01 23:36:23.429659 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:36:23.429628    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qlpzx_0d80334c-9fa2-413b-94ea-a4c353344f29/main/0.log"
Nov 01 23:36:40.625336 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:36:40.625298    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:37:10.625490 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:37:10.625448    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:37:36.409598 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:36.409569    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n7thn_af321a6a-4dd2-4105-9c8a-a5dbb8e6e3fb/global-pull-secret-syncer/0.log"
Nov 01 23:37:40.199868 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:40.199835    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hxbq6_a21a67f5-fb49-4af9-90b2-b68da8775bd6/konnectivity-agent/0.log"
Nov 01 23:37:40.626503 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:37:40.626458    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"
Nov 01 23:37:41.585114 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:41.585081    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-155.us-west-2.compute.internal_7f3192736cdd118e9e4731ee3fb56338/haproxy/0.log"
Nov 01 23:37:46.940066 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.939985    5031 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"]
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940335    5031 cpu_manager.go:414] "containerMap: removing container" podUID="f9a39cb3-50e2-42ff-8465-af0257488786" containerName="payload-processing"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940345    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a39cb3-50e2-42ff-8465-af0257488786" containerName="payload-processing"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940363    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940369    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940381    5031 cpu_manager.go:414] "containerMap: removing container" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940387    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940397    5031 cpu_manager.go:414] "containerMap: removing container" podUID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" containerName="payload-pre-processing"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940402    5031 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" containerName="payload-pre-processing"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940456    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e81b60c-148a-431a-b336-e0601f0bbf4d" containerName="istio-proxy"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940463    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="508f8a14-faac-452a-9b5c-1a31fe14ed3c" containerName="payload-pre-processing"
Nov 01 23:37:46.940575 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.940474    5031 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a39cb3-50e2-42ff-8465-af0257488786" containerName="payload-processing"
Nov 01 23:37:46.943427 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.943407    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:46.991689 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:46.991641    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"kube-root-ca.crt\""
Nov 01 23:37:47.037770 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.037727    5031 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wg4hr\"/\"default-dockercfg-76nmg\""
Nov 01 23:37:47.037770 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.037762    5031 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"openshift-service-ca.crt\""
Nov 01 23:37:47.040208 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.040183    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"]
Nov 01 23:37:47.068597 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.068562    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-proc\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.068786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.068621    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwsl\" (UniqueName: \"kubernetes.io/projected/d2a4129f-86f3-42c2-823a-0f3833dff6a1-kube-api-access-qhwsl\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.068786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.068732    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-podres\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.068786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.068756    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-lib-modules\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.068786 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.068777    5031 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-sys\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.169991 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.169946    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-proc\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170004    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwsl\" (UniqueName: \"kubernetes.io/projected/d2a4129f-86f3-42c2-823a-0f3833dff6a1-kube-api-access-qhwsl\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170064    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-proc\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170086    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-podres\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170117    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-lib-modules\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170203 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170143    5031 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-sys\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170409 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170201    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-podres\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170409 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170214    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-lib-modules\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.170409 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.170248    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2a4129f-86f3-42c2-823a-0f3833dff6a1-sys\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.223476 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.223399    5031 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwsl\" (UniqueName: \"kubernetes.io/projected/d2a4129f-86f3-42c2-823a-0f3833dff6a1-kube-api-access-qhwsl\") pod \"perf-node-gather-daemonset-zl59r\" (UID: \"d2a4129f-86f3-42c2-823a-0f3833dff6a1\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.256232 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.256191    5031 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:47.337592 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.337565    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-krsjt_f599a277-3519-4cbc-b948-7d6ada3183dc/kuadrant-console-plugin/0.log"
Nov 01 23:37:47.623095 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:47.623050    5031 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"]
Nov 01 23:37:47.630099 ip-10-0-138-155 kubenswrapper[5031]: W1101 23:37:47.630059    5031 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2a4129f_86f3_42c2_823a_0f3833dff6a1.slice/crio-5da3c3f0410a029ad966cb56dd1fe88930bef561300cb415a23a89b48d61b9f7 WatchSource:0}: Error finding container 5da3c3f0410a029ad966cb56dd1fe88930bef561300cb415a23a89b48d61b9f7: Status 404 returned error can't find the container with id 5da3c3f0410a029ad966cb56dd1fe88930bef561300cb415a23a89b48d61b9f7
Nov 01 23:37:48.619760 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:48.619729    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r" event={"ID":"d2a4129f-86f3-42c2-823a-0f3833dff6a1","Type":"ContainerStarted","Data":"3c92475f954d598c0bdb6061b0d2088ccccfaf9ffd61ba39d213054e60ede015"}
Nov 01 23:37:48.620187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:48.619786    5031 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r" event={"ID":"d2a4129f-86f3-42c2-823a-0f3833dff6a1","Type":"ContainerStarted","Data":"5da3c3f0410a029ad966cb56dd1fe88930bef561300cb415a23a89b48d61b9f7"}
Nov 01 23:37:48.620187 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:48.619880    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:48.727145 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:48.727087    5031 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r" podStartSLOduration=2.72707341 podStartE2EDuration="2.72707341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:37:48.725941053 +0000 UTC m=+4422.681980897" watchObservedRunningTime="2026-11-01 23:37:48.72707341 +0000 UTC m=+4422.683113284"
Nov 01 23:37:54.636328 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:54.636296    5031 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-zl59r"
Nov 01 23:37:55.855341 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:55.855314    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b56c5cc6f-zmpvr_735f3a97-3941-4569-8ca4-cb8beb72112b/metrics-server/0.log"
Nov 01 23:37:56.266679 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:56.266597    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-qcw24_16c03e15-6fbc-4efd-905f-ef9321873204/monitoring-plugin/0.log"
Nov 01 23:37:56.683008 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:56.682976    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d6wj_ac262171-cc91-4552-ad95-7af735ac5450/node-exporter/0.log"
Nov 01 23:37:57.088900 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:57.088876    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d6wj_ac262171-cc91-4552-ad95-7af735ac5450/kube-rbac-proxy/0.log"
Nov 01 23:37:57.502313 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:37:57.502245    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8d6wj_ac262171-cc91-4552-ad95-7af735ac5450/init-textfile/0.log"
Nov 01 23:38:01.697237 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:01.697203    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/prometheus/0.log"
Nov 01 23:38:02.105056 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:02.105023    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/config-reloader/0.log"
Nov 01 23:38:02.512850 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:02.512748    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/thanos-sidecar/0.log"
Nov 01 23:38:02.920772 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:02.920740    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/kube-rbac-proxy-web/0.log"
Nov 01 23:38:03.329859 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:03.329830    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/kube-rbac-proxy/0.log"
Nov 01 23:38:03.742388 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:03.742281    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/kube-rbac-proxy-thanos/0.log"
Nov 01 23:38:04.153208 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:04.153179    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b432e902-88e9-414d-9203-b9215c810eda/init-config-reloader/0.log"
Nov 01 23:38:04.568107 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:04.568039    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7lctl_c780c544-2eef-4d4c-8b46-b3f079ef2e72/prometheus-operator/0.log"
Nov 01 23:38:04.969839 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:04.969759    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7lctl_c780c544-2eef-4d4c-8b46-b3f079ef2e72/kube-rbac-proxy/0.log"
Nov 01 23:38:10.106161 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:10.106120    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59b78b6bf6-xwnhc_501fa75d-cc43-478c-8d1c-d85b342bf25e/kube-auth-proxy/0.log"
Nov 01 23:38:10.306782 ip-10-0-138-155 kubenswrapper[5031]: I1101 23:38:10.306732    5031 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-2zqr6_bc9e3876-1e29-4e71-af35-638422d47e53/istio-proxy/0.log"
Nov 01 23:38:10.626640 ip-10-0-138-155 kubenswrapper[5031]: E1101 23:38:10.626600    5031 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5bshs" containerName="extractor"